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THE  EXPERIMENTAL  TECHNOLOGY  INCENTIVES  PROGRAM 


The  Experimental  Technology  Incentives  Program  (ETIP)  of  the 
National  Bureau  of  Standards  pursues  an  understanding  of  the 
relationships  between  government  policies  and  technology-based 
economic  growth.  The  pursuit  of  this  objective  is  based  on 
three  premises: 

• Technological  change  is  a significant  contributor  to 
social  and  economic  development  in  the  United  States. 

• Federal,  State,  and  local  government  policies  can  influ- 
ence the  rate  and  direction  of  technological  change. 

• Current  understanding  of  this  influence  and  its  impact 
on  social  and  economic  factors  is  incomplete. 

ETIP  seeks  to  improve  public  policy  and  the  policy  research 
process  in  order  to  facilitate  technological  change  in  the 
private  sector.  The  program  does  not  pursue  technological 
change  per  se.  Rather,  its  mission  is  to  examine  and  experiment 
with  government  policies  and  practices  in  order  to  identify 
and  assist  in  the  removal  of  government-related  barriers  and 
to  correct  inherent  market  imperfections  that  impede  the  innova- 
tion process. 

ETIP  assists  other  government  agencies  in  the  design  and  conduct 
of  policy  experiments.  Key  agency  decisionmakers  are  intimately 
involved  in  these  experiments  to  ensure  that  the  results  are 
incorporated  in  the  policymaking  process.  ETIP  provides  its 
agency  partners  with  both  analytical  assistance  and  funding 
for  the  experiments  while  it  oversees  the  evaluation  function. 

Eecause  all  government  activities  potentially  can  influence 
the  rate  and  direction  of  technological  change,  ETIP  works 
with  a wide  variety  of  agencies,  including  those  that  have 
regulatory,  procurement,  R&D,  and  subsidy  responsibilities. 
Programs  are  currently  underway  with  the  General  Services  Adminis 
tration,  Food  and  Drug  Administration,  Veterans  Administration, 
Securities  and  Exchange  Commission,  Department  of  Energy,  Environ 
mental  Protection  Agency,  Occupational  Safety  and  Health  Adminis- 
tration, and  other  Federal  agencies  as  well  as  various  State 
and  local  agencies. 

The  accompanying  report  was  prepared  under  contract.  Statements 
contained  in  this  document  represent  the  views  of  the  originating 
organization  and  do  not  necessarily  reflect  those  of  the  National 
Bureau  of  Standards. 
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PREFACE 


This  paper  assesses  an  experiment  conducted  by  the  Experimental 
Technology  Incentives  Program  (ETIP)  in  conjunction  with  the  Nuclear 
Regulatory  Commission  (NFC)  . This  assessment  was  carried  out  as  part  of 
analytic  support  contracts  awarded  to  The  Urban  Institute  and  the  Performance 
Development  Institute^-  by  ETIP. 

The  purpose  of  the  assessment  is  twofold: 

(1)  To  review  the  experiment  and  examine  whether  further  detailed 
evaluation  is  warranted,  and 

(2)  To  determine  what  lessons  can  be  learned  about  the  conduct  of 
ETIP  experiments  in  general. 

The  underlying  intent  throughout  has  been  to  uncover  the  facts  about  the 
experiments  so  that  an  objective,  accurate  assessment  could  be  made. 
Information  for  the  paper  was  gathered  by  reviewing  available  documenta- 
tion on  the  project  and  on  the  organizations  involved,  and  by  interviewing 
individual  participants.  The  original  designers  of  the  experiment  have  been 
very  helpful  in  this  regard.  The  results  of  the  study  represent  the  views 
of  the  authors  and  are  meant  as  a critique  of  method,  not  the  designers. 

The  lessons  learned  from  the  assessment  should  help  improve  the  design  of 
future  ETIP  experiments. 

ETIP's  goal  is  to  develop  policy  guidelines  relating  to  the  impact  of 
government  on  industry  and  on  technological  change  as  well  as  a body  of 
knowledge  necessary  for  their  effective  use.  To  achieve  these  goals,  ETIP 
has  been  carrying  out  experiments  with  regulatory  agencies  to  develop  a 

1.  Department  of  Commerce  Contract  #7-35822  to  The  Urban  Institute, 
October  1,  1977  to  December  31,  1978.  Department  of  Commerce  Contract 
#E0-78-3603  to  the  Performance  Development  Institute  for  October  1,  1978 
to  December  31,  1979. 
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knowledge  base  and  policy  recommendations  on  how  regulatory  processes  can 
be  changed  to  bring  about  the  desired  commercial  impact  and  to  improve  the 
environment  for  technological  innovation.  Under  its  contract  with  ETIP, 

The  Urban  Institute  and  the  Performance  Development  Institute  have  provided 
analytic  and  data  collection  support. 

One  assumption  underlying  ETIP  experimentation  is  that  the  rate  and 
direction  of  private  sector  activity  is  influenced  by  the  regulatory  process. 
ETIP  also  assumes  that  innovation  plays  a leading  role  in  economic  and  social 
growth.  A final  assumption  is  that  administrative  experimentation  will  lead 
to  better  understanding  and  control  of  the  regulatory  processes  involved. 

A major  focus  of  ETIP's  work  is  on  the  regulatory  development  process. 
This  process  often  involves  lengthy  time  delays,  known  as  "regulatory  lag." 
One  example  of  the  lag  problem  exists  in  the  nuclear  regulatory  field,  where 
the  Nuclear  Regulatory  Commission  (NRC)  adopts  standards  for  the  licensing 
of  nuclear  reactors. 

The  NRC's  mandate  is  to  assure  that  non-military  uses  of  nuclear 
materials  and  facilities  are  consistent  with  the  laws  governing  the  public 
health  and  safety,  environmental  quality,  national  security  and  antitrust. 

The  NRC  implements  this  mandate  through  the  licensing  of  nuclear  materials 
and  facilities  and  the  conduct  of  programs  to  ensure  compliance.  Standards 
for  nuclear  plants  are  included  in  regulations  or  regulatory  guides  for 
companies  seeking  nuclear  power  plant  licenses. 

ETIP's  selection  of  the  NRC  and  the  standards-setting  process  was  based 
upon  public  concern  with  energy  problems  and  with  the  costly  delay  in  the 
licensing  of  nuclear  power  plants.  The  absence  of  standards  for  nuclear 
power  plants  was  thought  to  contribute  to  such  delays.  Thus  the  goal  of  the 
ETIP  project  described  in  this  report  was  to  explore  ways  to  hasten  both  the 
establishment  of  standards  and  their  adoption  by  the  NRC. 
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EXECUTIVE  SUMMARY 


In  March  1974,  the  Experimental  Technology  Incentives  Program  (ETIP) 
working  in  conjunction  with  the  Nuclear  Regulatory  Commission  (NRC)  and  the 
American  National  Standards  Institute  (ANSI)  completed  a project  plan  to 
experiment  with  accelerating  the  initial  drafting  of  nuclear  power  plant 
standards  by  drafting  subcommittees.  These  drafting  subcommittees  were 
staffed  by  volunteers  from  private  industry  and  government,  as  part  of  the 
voluntary  standards  development  process.  The  plan  outlined  the  following 
activities : 

1.  ETIP  was  to  prepare  a project  plan  to  test  the  following  four 
changes  from  the  normal  standard  drafting  process  for  three  or 
four  standards  (each  standard  was  to  be  one  experiment): 

— Hiring  a full-time  working  group  chairman, 

— Providing  consultant  services  for  data  collection/correlation, 

— Providing  clerical/ technical  editing  services, 

— Conducting  extended  working  meetings,  possibly  up  to  14  days. 

2.  ETIP  would  obligate  funds  for  the  experiments  for  NRC  use. 

3.  NRC  would  contract  with  ANSI  to  conduct  experiments. 

4.  ANSI  would  hire  a project  manager  to  coordinate  the  individual 
experiments  and  monitor  the  effects  of  the  projects  on  the 
development  of  standards. 

5.  Each  experiment  would  differ  in  terms  of  the  variables  tested, 

(see  //I  above)  , although  the  committees  could  choose  the  variables 
to  be  tested. 

6.  ETIP  would  contract  with  an  evaluator  to  conduct  a project  evalu- 
ation, to  assess: 

— whether  standard  development  schedules  were  met, 

— quality  of  standards  produced, 

— whether  a more  effective/efficient  use  of  working  group 
members  resulted, 

— whether  drafting  time  reduction  offset  project  costs. 

7.  ETIP  would  conduct  further  research  as  needed,  depending  on  results. 
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The  standard-drafting  experiments  were  expected  to  accelerate  the  produc- 
tion of  draft  standards,  which  in  turn  was  expected  to  accelerate  the  produc- 
tion of  final  standards  and  their  ultimate  adoption  by  the  NRC.  If  the  project 
was  successful,  multiple  benefits  (e.g.,  reduction  of  nuclear  reactor  licensing 
delays  and  industry  costs)  were  anticipated. 

The  purpose  of  this  report  is  to  describe  and  assess  the  experiments 
conducted,  to  report  the  lessons  learned  from  the  project,  and  to  review 
the  usefulness  of  further  research.  It  is  based  on  an  open,  objective  inquiry 
about  the  facts  of  the  project.  To  the  extent  possible,  the  findings  from 
the  assessment  are  included  in  the  report  so  that  readers  can  understand  the 
project  and  draw  their  own  conclusions. 

A.  THE  EXPERIMENTAL  INTERVENTION 

The  actual  intervention  conducted  varied  from  the  initial  plan.  ANSI 
hired  two  contractors  to  conduct  the  experiments.  Battelle-Northwest 
(contractor  #1)  undertook  experiments  with  three  separate  subcommittees. 

The  variables  tested  included: 

— Use  of  a concentrated  five-day  working  meeting  to  draft 
the  standard  (instead  of  several  one  or  two-day  sessions, 
held  over  many  months) . 

— Premeeting  preparation  of  a working  draft  outline. 

— Provision  of  an  executive  secretary/ coordinator . 

— Provision  of  clerical/duplicating  support. 

— Payment  of  working  group  chairmen. 

— Payment  of  travel  and  living  expenses  of  work  group  members. 

— Holding  of  preliminary  organizational  meetings  before  the 
five-day  working  session. 

— Limiting  working  group  membership  to  10  members. 
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Anderson,  the  second  contractor,  limited  his  experiment  to  the  provision 
of  secretarial  support  to  the  subcommittees  involved.  Each  of  the  contractors 
submitted  reports  on  their  project  experiments,  but  these  were  not  intended 
to  be  comprehensive  impact  evaluations.  Impact  evaluations  were  planned  by 
project  designers,  but  were  not  funded  due  to  complications  in  contracting  and 
program  changes  in  ETIP. 


B.  ASSESSING  THE  NUCLEAR  REGULATORY  EXPERIMENT 

Three  criteria  were  identified  in  the  project  plan  as  important  in 
evaluating  project  success: 

• Length  of  time  required  to  produce  the  standard  (as  compared 
to  "normal"  time  requirements)  . 

e Quality  of  the  standard  produced. 

• Cost . 

The  Battelle  and  Anderson  reports  provided  some  information  on  these 
criteria  even  though  the  two  reports  were  not  intended  to  be  comprehensive 
evaluations.  The  following  sections  review  the  data  available  on  each  criterion. 

1 . TIME 

In  the  Battelle  experiment,  the  time  required  to  draft  the  three  standards 
was  considered  to  be  faster  than  the  norm  for  the  production  of  draft  standards. 
While  the  Anderson  report  provided  insufficient  detail  to  assess  the  results 
reported,  that  report  also  concluded  that  the  experiment  helped  to  speed  pro- 
duction of  draft  standards. 

Neither  of  the  reports  associated  with  the  experiment  traced  the  time 
required  for  the  draft  standard  to  proceed  through  the  remainder  of  the 
standards  development  process.  This  trace  would  have  been  needed  to  determine 


the  overall  time  required  to  produce  the  finished  adopted  standard.  Original 
plans  had  discussed  doing  this.  However,  even  if  the  traces  had  been  carried 
out,  a comparison  to  normal  conditions  would  have  been  difficult  since  it 
appears  that  little  data  is  available  on  the  normal  time  required  to  produce 
a standard. 

2.  QUALITY 

No  precise  criteria  exist  for  evaluating  the  quality  of  standards  produced 
through  the  voluntary  standards  process,  and  none  of  the  reports  associated 
with  the  experiment  attempted  to  assess  the  quality  of  the  standards  produced. 
The  evaluation  study  which  had  been  projected  to  assess  standard  quality  was 
never  contracted  out  . 

3.  COST 

Neither  report  indicated  exactly  how  the  money  was  spent  or  allocated 
among  the  experimental  variables.  It  was  thus  impossible  to  identify  the  cost 
of  implementing  the  individual  variables. 


C.  LESSONS  LEARNED 

Three  experimental  changes — provision  of  secretarial  assistance,  clerical/ 
duplicating  support,  and  the  five-day  work  week — were  reported  to  have  acceler- 
ated the  initial  drafting  process.  The  evidence  supporting  these  claims  is  weak, 
however,  and  the  effect  of  these  changes  remains  unknown.  Nevertheless,  there 
were  important  lessons  learned  from  the  experiment,  and  these  relate  to  the 
experimental  process  rather  than  to  the  outcomes  of  the  experiments  conducted. 
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1 . THE  KNOWNS 


There  were  a number  of  problems  in  the  experiment: 

• As  a result  of  having  to  transfer  funds  through  the  NRC  and 
ANSI,  ETIP  was  unable  to  maintain  close  control  over  the 
design,  development  and  implementation  of  the  experiments. 

• Because  of  planning  and  funding  delays,  the  evaluation  origin- 
ally planned  to  assess  project  outcomes  and  impacts  did  not 
take  place. 

• Information  provided  by  contract"’’0  on  key  project  variables 
and  outcomes  was  incomplete. 

• The  design  initially  chosen  for  the  project  was  limited  in 
the  amount  of  information  it  could  actually  provide. 

• The  initial  design  did  not  clearly  lay  out  how  the  proposed 
experiment  fit  into  the  major  ETIP  research  needs,  or  into 
the  overall  standards  development  process. 

• No  background  information  was  collected  at  the  outset  to  ini- 
tially check  the  validity  of  the  experimental  assumptions.  A 
questioning  of  these  assumptions  by  ETIP  staff  might  have  per- 
mitted a more  defensible  targeting  of  the  experimental  resources, 
and  questions  about  the  project's  relevancy  could  have  been  better 
addressed  both  during  the  experiment  and  afterward. 


2.  THE  UNKNOWNS 

Because  of  the  problems  with  experimental  design  and  the  lack  of  a com- 
prehensive, follow-on  evaluation,  much  remains  uncertain  at  this  time  about 
the  impacts  of  the  experimental  changes.  Little  or  no  information  was 
collected  on  such  factors  as: 

• average  time  for  standards  drafting  and  processing, 

• the  role  of  existing  standards  in  the  plant  licensing  process, 

• quality  of  the  standards  produced, 

• cost  of  introducing  the  individual  innovations,  compared 
with  normal  production  costs,  and 

• the  details  of  the  experiments  conducted 

It  would  be  difficult  and  costly  to  obtain  information  on  these  factors  at 
this  point  in  the  project. 
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3.  USEFULNESS  OF  FURTHER  STUDY 

ETIP  has  several  options  relative  to  the  conduct  of  further  study  on 
the  voluntary  standards  program: 

Option  1:  Terminate  the  research  at  the  conclusion  of  the  present 

review. 

Option  2:  Fund  a limited  research  effort  to  gather  additional  back- 

ground information  about  the  experiments  conducted  and  their  impacts  (e.g., 
the  costs  of  the  individual  innovations,  the  quality  of  the  standards 
produced)  as  well  as  about  the  "normal"  development  process  and  time  require- 
ments in  the  nuclear  standards  area,  with  which  to  compare  the  experimental 
results . 

Option  3:  Fund  a substantial  research  effort  to  identify  the  role  of 

standards,  if  any,  in  accelerating  nuclear  plant  licensing,  using  a research 
design  and  a contracting  process  which  will  address  the  difficulties  exper- 
ienced in  the  initial  experiment.  Assuming  that  standards  are  found  to  be  a 
substantial  factor  in  accelerating  nuclear  plant  licensing,  ETIP  could  consider 
the  usefulness  of  investigating  how  and  whether  the  voluntary  standards  process 
can  be  successfully  accelerated  to  develop  those  standards. 

Option  4:  Fund  an  evaluability  assessment  of  the  usefulness  of  conducting 

further  study  of  the  voluntary  standards  process  and  the  impact  of  standards  on 
nuclear  power  plant  licensing.  This  would  involve  collecting  sufficient  back- 
ground information  about  the  proposed  research  that  the  costs  and  likely  results 
and  benefits  from  pursuing  the  various  research  options  could  be  assessed  prior 
to  committing  funds  to  any  particular  research  design. 

It  is  the  conclusion  of  this  assessment  that  if  further  research  is  con- 
templated, Option  4 is  the  most  highly  recommended.  The  conduct  of  an  evalu- 
ability assessment  would  provide  ETIP  with  guidance  as  to  the  areas  where 
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further  research  is  desirable  and  likely  to  be  fruitful.  Option  1 is  the 
alternative  best  choice  if  sufficient  resources  are  not  available  to  make  it 
worthwhile  to  undertake  an  evaluability  assessment. 

4.  SUMMARY 

While  the  drafting  process  was  accelerated,  the  research  design  and  the 
lack  of  sufficient  data  to  assess  the  experimental  assumptions  at  the  design 
stage  produced  an  experiment  that  was  unable  to  demonstrate  whether  the 
acceleration  of  the  drafting  stage  accelerated  the  entire  standards  development 
process  and  subsequent  NRC  endorsement. 


I.  INTRODUCTION 


In  1975,  the  Experimental  Technology  Incentives  Program  (ETIP)  undertook 
an  experiment  to  acccelerate  the  establishment  of  standards  and  their  adoption 
as  regulatory  guides  by  the  Atomic  Energy  Commission  (now  the  Nuclear  Regula- 
tory Commission,  or  NRC) . This  particular  experiment  was  selected  because  of 
political  interest  at  that  time  with  reducing  nuclear  plant  licensing  delays. 
Licensing  delays  were  attributed  in  part  to  the  absence  of  nuclear  power  plant 
standards.  Accelerating  the  development  of  nuclear  standards  was  expected  to 
reduce  such  delays.  Figure  1 below  depicts  the  assumptions  underlying  the 
experiment . 


ETIP/NRC 

Experiment- 


Draft  standards 
are  produced  — 
more  rapidly 


Final  standards 
-►  are  produced  and- 
are  adopted  by 
NRC  more  rapidly 


Standards  are 
-►used  by  NRC 
and  industry 


Regulatory  review 
required  for 
reactor  licensing 
is  reduced  for 
components  where 
standards  are  used 


Reduction  in  reactor 
licensing  delays 

Reduction  in  industry 
and  government  costs 

Stimulation  of  industry 
investment  in  technology 

Stimulation  of  timely 
implementation  of  technology 

Reduction  of  human  error 
and  increased  reactor  safety 


FIGURE  1:  EXPERIMENTAL  EXPECTATIONS 
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The  project  was  viewed  as  only  the  first  of  a series  of  ETIP  projects  to 
examine  the  relationship  between  voluntary  or  industry-based  standards  and 
regulatory  processes. 

The  purpose  of  this  report  is  to  describe  the  experiment  conducted  and 
its  findings,  and  to  depict  how  this  particular  experiment  fit  into  the  over- 
all scheme  of  ETIP  experimentation.  It  follows  up  on  and  in  part  relies  on 
information  gathered  in  an  ETIP  case  study  of  the  experiment  conducted  in  1976 
and  reported  in  January,  1977.^  This  report  also  purports  to  assess  the 
overall  effectiveness  of  the  experiment  and  to  pinpoint  the  problems  encoun- 
tered and  how  such  problems  might  be  avoided  in  the  future,  although  it  is  not 

2 

an  evaluation  of  the  experiment  per  se . Finally,  the  report  traces  the 
standards  produced  as  part  of  the  experiments,  and  presents  options  for  ETIP's 
consideration  in  terms  of  further  study  which  might  be  undertaken. 

Information  for  this  report  was  gathered  by  reviewing  available  docu- 
mentation on  the  project  and  on  the  organizations  involved,  and  by  inter- 
viewing individual  participants.  Individuals  from  the  American  National 
Standards  Institute  (ANSI),  the  NRC,  the  Experimental  Technology  Incentives 
Program  (ETIP),  and  the  National  Bureau  of  Standards  (NBS)  were  interviewed, 
either  in  person  or  by  telephone.  (Appendix  A lists  the  individuals  inter- 
viewed and  the  major  documentation  reviewed.) 


1.  Garrity,  Stephen,  "A  Review  of  the  Nuclear  Regulatory  Experiment," 

unpublished  draft,  Washington,  D.C.:  Experimental  Technology  Incentives 

Program,  National  Bureau  of  Standards,  January  11,  1977.  See  Appendix  D. 

2.  ETIP  had  planned  to  contract  out  an  evaluation  to  trace  the  impacts 
of  the  experiment.  Planning  for  the  evaluation  occurred  after  the  experiment 
had  been  initiated,  however,  and  although  money  for  the  evaluation  was  obli- 
gated, no  proposals  submitted  for  the  evaluation  were  funded. 


I 
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The  report  is  organized  into  several  sections.  Section  II  describes 
the  context  within  which  the  experiment  was  conducted:  the  main  players  or 

"stakeholders,"  and  their  established  roles  in  the  voluntary  standards  process. 
Section  III  outlines  the  expectations  held  by  the  project  planners  when  the 
experiment  was  initially  designed:  the  assumptions  which  the  experiment  was 

intended  to  test.  Section  IV  describes  the  project  activities  as  they  were 
reported  by  project  documentation  and  the  individuals  interviewed,  and  the 
products  produced.  Section  V then  contrasts  what  was  expected  to  occur  with 
what,  in  fact,  did  occur,  and  describes  what  was  learned  from  the  experiment. 
The  final  section  addresses  major  findings,  what  remains  unknown,  options  for 
further  study,  and  "lessons  learned"  from  the  experiment  applicable  to  the 
design  of  future  experiments. 


II.  CONTEXT  OF  THE  EXPERIMENT:  AN  OVERVIEW  OF  THE 

VOLUNTARY  STANDARDS  PROCESS 


The  primary  function  of  government  agencies,  such  as  the  NRC,  is  to  im- 
plement Congressional  legislation.  A crucial  NRC  mandate,  for  example,  is  to 
ensure  "a  high  level  of  public  health  and  safety  and  environmental  protection 
in  the  design,  construction,  and  operation  of  nuclear  power  reactors."^  One 

means  for  carrying  out  this  mandate  is  through  NRC  promulgation  or  adoption  of 
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standards  for  the  licensing  of  nuclear  reactors.  The  NRC  then  includes  these 

standards  in  its  regulatory  program  as  regulations  or  regulatory  guides  for 

3 

companies  interested  in  obtaining  nuclear  power  plant  licenses. 

The  NRC's  standards  have  been  viewed  as  serving  the  following  major  pur- 
poses, to: 

1.  define  information  required  from  an  applicant  for  license, 

2.  define  acceptable  levels  of  safety  and  environmental  protection, 

3.  provide  inspection  and  enforcement  criteria, 

4.  limit  contested  issues  in  hearings. 


1.  Testimony  of  Lester  Rogers,  Director  of  Regulatory  Standards,  U.S. 
Atomic  Energy  Commission,  "Hearings  on  Reactor  Safety  before  the  Joint  Commit- 
tee on  Atomic  Energy,"  September  26,  1973,  p.  1. 

2.  Standards  can  be  defined  as  written  specifications  (codes  and  crite- 
ria) establishing  minimum  requirements  for  the  technical  design,  construction 
and  performance  of  specific  products  or  for  the  conduct  of  specific  activities. 

3.  As  noted  in  the  Rogers'  testimony,  regulations  "set  forth  both  general 
and  specific  requirements  which  must  be  met"  by  a licensing  applicant.  Regula- 
tory guides,  on  the  other  hand,  provide  "guidance  to  applicants  and  licensees 
on  ways  acceptable  to  the  regulatory  staff  of  complying  with  the  general  and 
specific  regulations"  (see  page  2).  However,  compliance  with  the  regulatory 
guides  is  not  required,  if  an  alternative  and  acceptable  way  of  complying  with 
the  regulations  is  proposed. 
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5.  inform  the  public, 

6.  provide  models  and  methodology  for  assessing  levels  of  safety, 
safeguards,  and  environmental  protection,  and 

7.  help  provide  a basis  for  standardization  of  plants. 

The  NRC  can  develop  its  own  standards  for  inclusion  in  the  regulations 
and  regulatory  guides.  Alternatively,  when  it  recognizes  the  need  for  a 
standard,  the  NRC  may  choose  to  adopt  or  adapt  an  existing  standard  if  the 
needed  standard  already  has  been  developed  by  an  interest  group  (e.g.,  a par- 
ticular industry  or  a consumer  group) . Where  the  needed  standard  has  not  yet 
been  developed,  the  NRC  can  request  that  a standard  be  developed  by  submitting 
its  request  to  a group  such  as  the  American  National  Standards  Institute  (ANSI) . 
That  request  would  be  handled  by  the  appropriate  committee  within  ANSI — in 
this  case  of  nuclear  standards,  the  Nuclear  Standards  Management  Board  (NSMB)  . 
The  NSMB  would  be  responsible  for  coordinating  the  standard  development  effort, 
the  end  product  of  which  would  be  submitted  to  the  NRC  (as  was  done  in  the 
experiment  discussed  herein) . 


A.  ANSI  AND  THE  VOLUNTARY  STANDARDS  PROCESS 

Thousands  of  technical,  product-related  standards  have  been  generated  in 

the  United  States;  over  8000  have  been  adopted  by  ANSI  as  American  National 
2 

Standards.  Many  such  standards  have  been  generated  by  industry  on  a 
voluntary  basis  (as  opposed  to  government-developed  standards  imposed  through 
regulation).  Over  400  trade  associations,  professional  societies,  and  other 
similar  interest  groups  (including  consumer  groups)  have  sponsored  and  coordi- 
nated the  standards-development  efforts.  (Standard-setting  may  comprise  from 


1.  Testimony  of  Lester  Rogers,  p.l. 

2.  Interview  with  ANSI  official. 


10-30  percent  of  the  typical  association's  activity.)  This  standards  develop- 
ment process  is  known  as  the  industry-based  or  voluntary  standards  system. 

ANSI  is  a federation  of  standards-developing  and  standards-using  organi- 
zations comprising  approximately  900  companies  and  200  trade,  technical,  labor, 
and  consumer  organizations.  ANSI's  main  functions  include  managing  and  coor- 
dinating the  voluntary  development  of  national  standards  by  its  participating 
members,  and  approving  such  standards  as  American  National  Standards  if  they 
meet  ANSI's  consensus  requirements.  ANSI's  standards  development  process 
emphasizes  procedural  fairness,  a balancing  of  the  interests  represented  at 
the  drafting  and  review  stages,  and  a consensus  approach  to  standard  adoption. 
According  to  an  ANSI  report: 

These  (consensus)  requirements  ensure  that  all  substantially 
affected  interests  have  had  an  opportunity  to  participate  in  the 
standard's  development  or  to  comment  on  its  provisions  and  that 
the  standard  has  achieved  general  recognition  and  acceptance  for 
use.  Organizations  that  submit  standards  to  ANSI  for  approval 
must  supply  evidence  that  these  and  related  consensus  criteria 
have  been  met.  ANSI  then  conducts  independent  verification  of 
the  adequacy  of  consensus  before  acting  on  approval. ^ 

Under  the  ANSI  consensus  approach,  something  greater  than  a majority  but  less 
than  unanimity,  is  required.  Any  substantive  objection  to  the  standard  must 
be  reviewed  and  responded  to  by  the  drafting  group.  If  the  objecting  party 
is  not  satisfied  by  the  response,  the  objection  accompanies  the  draft  standard 
through  the  entire  review  process  to  publication.  If  the  substantive  objec- 
tions are  numerous  and  unresolvable , the  standard  must  be  withdrawn. 

ANSI's  public  review  or  "canvassing"  process  can  involve  sending  a pro- 
posed standard  up  to  14,000  individuals  and  groups,  including  media  outlets 
and  governmental  agencies.  Standards  which  survive  the  review  process  without 

1.  "ANSI  Progress  Report,  1977,"  New  York:  American  National  Standards 
Institute,  March  1977,  p.  1. 
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major  substantive  objections  (or  where  the  substantive  objections  are  in  the 
main  resolved  through  standard  revision)  are  adopted  as  American  National 
Standards  . 

ANSI  coordinates  the  standard- se tting  activities  of  its  member  groups. 

It  is  organized  into  a number  of  committees  handling  particular  subject  areas. 
For  example,  the  Nuclear  Standards  Management  Board  (NSMB)  is  responsible 
for  overseeing  the  nuclear  standards  program,  including  policy,  financing, 
and  recruiting Governmental  or  industry  requests  for  particular 
standards  (e.g.,  from  the  NRC)  are  funnelled  through  individual  ANSI  commit- 
tees such  as  the  NSMB.  Generally  at  least  one  federal  government  employee 
is  a member  of  such  ANSI  committees. 

ANSI  is  dependant  upon  membership  dues  and  the  sale  of  its  published 
standards  for  its  main  support,  although  it  does  receive  some  special  project 

support.  Less  than  3 percent  of  its  operating  budget  comes  from  government 

2 

sources  in  the  form  of  grants  for  specified  programs.  One  ANSI  employee 
indicated  that  the  government  estimates  the  voluntary  standardization  process 
to  cost  over  $500  million  per  year  in  volunteer  labor. 

Figure  2 below  depicts  the  ANSI  standard  development  process  in  summary 
form,  showing  the  participants  in  the  process  and  their  respective  roles. 

Hie  process  is  described  in  greater  detail  in  the  following  pages. 


1.  Ibid,  p.  12. 

2.  Ibid,  pp.  11-13. 
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FIGURE  2:  VOLUNTARY  STANDARDS  PROCESS 

UTILIZED  BY  NRC 


B.  STANDARDS  PROCESS  IN  DETAIL 


Interviews  revealed  that  in  the  usual  standard-setting  process,  the 
interest  group  which  has  "responsibility"  over  the  standard  area  and  which 
follows  ANSI-prescribed  consensus  procedures  works  in  cooperation  with  ANSI. 

The  typical  standard  development  pattern  was  described  by  ANSI  staff  and  other 
participants  as  follows : 

1.  An  entity  ( individual , group,  government)  proposes  to  an  interest 
group  or  to  ANSI  that  a standard  is  needed  for  an  activity.  If  pro- 
posed to  ANSI,  the  appropriate  ANSI  committee  refers  the  standard  to 
the  appropriate  interest  group  [e.g.,  the  American  Nuclear  Society 
(ANS)]  . The  interest  group  is  then  responsible  for  seeing  that  the 
draft  standard  is  developed  in  line  with  ANSI's  procedureal  require- 
ments . 

2.  The  interest  group  refers  the  matter  to  the  committee  responsible  for 
the  particular  activity  involved.  If  no  committee  is  appropriate,  the 
group  notifies  its  membership,  requesting  those  interested  to  volunteer 
to  form  a committee  to  draft  the  standard.  If  a sufficient  number  re- 
spond, the  committee  members  are  selected  from  those  responding.  (If 
too  few  respond,  no  action  on  the  proposal  is  taken.)  An  effort  is 
generally  made  to  balance  the  various  interests  on  the  committee.  A 
government  official  may  or  may  not  serve  on  the  committee. 


3.  A subcommittee  chair  and  members  are  selected  as  the  writing  group 

for  the  committee  to  prepare  an  initial  draft  of  the  standard.  Draft 
preparation  typically  involves  the  following  steps: 

a.  The  first  one-day  meeting:  The  initial  meeting  is  mainly 

organizational,  to  discover  interests,  information  needs,  and 
how  to  organize  knowledge.  Sections  to  be  drafted  may  be 
assigned  to  individual  members. 

b.  Second  meeting  held  several  months  later:  Section  drafts  are 

probably  not  completed;  discussion  focuses  on  resolving 
various  interest  conflicts  and  substantive  issues. 

c.  Third  meeting  held  some  months  later:  Section  drafts  cir- 

culated in  advance  are  discussed,  issues/interests  worked  on. 

d.  Additional  meetings  are  held.l  2 

e.  The  working  group  votes  on/accepts  the  draft. 

4.  The  draft  standard  is  submitted  to  the  full  committee  for  voting. 
(Objections  are  responded  to  by  the  writing  group.) 

5.  The  draft  standard  (if  accepted  by  the  full  committee)  is  submitted 
to  the  interest  group  membership  as  a whole  for  voting.  (Objections 
are  responded  to  by  the  writing  group.) 

6.  The  draft  standard  (if  accepted  by  the  interest  group)  is  submitted 
to  ANSI  for  canvassing  (submission  for  public  review  and  comment  by 
the  14,000  individuals  and  organizations  on  its  review  list.)  Com- 
ments are  returned  to  the  writing  group  for  response. 


1.  The  number  of  meetings  required  and  the  length  of  time  between 
meetings  typically  depends  on  such  factors  as  the  status  of  knowledge  about 
the  activity  (whether  studies  need  to  be  conducted  or  the  activity  has  been 
studied  and  knowledge  is  fairly  well  codified) , the  extent  of  controversy 
about  the  activity,  practices  of  the  interest  group  involved,  use  or  non-use 

of  the  consensus  approach,  the  urgency  of  need  for  the  standard,  etc.  Meetings 
are  generally  several  months  apart,  since  they  must  be  fitted  in  on  a volunteer 
basis  on  time  contributed  by  the  individuals  involved  and  expenses  underwritten 
by  the  companies  or  governmental  entities  employing  the  individuals.  An  inter- 
est group  staff  member  may  or  may  not  be  assigned  to  provide  administrative, 
logical,  and  clerical  support. 

2.  An  interest  group  typically  has  certain  required  voting  procedures 
as  well  as  an  appeals  process  at  each  voting  level.  For  example,  the  voting 
procedures  may  require  that  80%  of  the  members  vote,  and  that  70%  of  the  votes 
be  positive.  In  addition,  where  the  voluntary  consensus  process  is  used,  the 
rules  generally  require  that  all  objections  be  examined  and  responded  to,  and 
a rationale  provided  for  why  the  objection  is  being  overruled  if  the  objecting 
party  is  not  satisfied  with  the  writing  group's  response.  If  the  objecting 
member  is  dissatisfied,  the  member  may  appeal  the  group's  decision,  with  the 
final  level  of  appeal  being  a decision  by  the  board  of  directors  for  the 
particular  interest  group.  Rejection  at  any  level  throws  the  standard  back 

to  the  writing  group  to  respond  to  the  objections  raised. 
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7.  The  writing  group  responds  to  the  comments  by  making  revisions  or 
providing  a rationale  for  why  revision  is  not  appropriate. 

8.  ANSI  reviews  the  comments  and  responses  and  determines  whether  ANSI 
consensus  criteria  are  met,  permitting  standard  adoption. 1 

9.  ANSI  adopts  and  publishes  the  standard  as  an  American  National 
Standard  (if  consensus  criteria  are  met).  Interested  parties 
(including  the  government)  may  purchase  and  use  the  standard. 

10.  The  pertinent  government  agency  (e.g.,  the  NRC)  reviews  the  ANSI 
standard.  (An  optional  step  for  the  agency.) ^ 

11.  The  agency  endorses  the  standard,  fails  to  endorse  the  standard, 
or  endorses  the  standard  with  exceptions.  If  endorsed  or  endorsed 
with  exceptions,  the  standard  is  incorporated  as  part  of  a regula- 
tory guide  or  as  a regulation.  (If  not  endorsed,  the  agency  may 
request  revisions  or  draft  its  own  standard.) 

12.  Those  regulated  (industry  groups,  manufacturers,  licensees)  are 
required  (if  the  standard  is  made  a regulation)  or  encouraged  (if 
the  standard  is  incorporated  in  a regulatory  guide)  to  use  the 
standard  endorsed  by  the  government  agency. 
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1.  ANSI's  consensus  approach  is  described  on  page  5,  i.e.,  the  require- 
ments of  greater  than  majority  approval  and  subcommittee  response  to  all 
substantive  objections  raised. 

2.  The  NRC's  Director  of  Regulation  has  made  a commitment  that  an  ANSI- 
approved  nuclear  standard  will  be  reviewed  and  action  taken  to  implement  the 
standard  within  90  days,  or  ANSI  will  be  informed  why  adoption  did  not  occur. 


III.  THE  EXPERIMENTAL  INTERVENTION 


The  goal  of  this  project  was  to  accelerate  the  establishment  of  standards 
and  their  adoption  by  the  NRC  as  regulatory  guides.  The  expectation  of  the 
project  planners  was  that  by  altering  the  conditions  und^”  which  the  interest 
group  drafting  subcommittees  wrked,  the  time  needed  to  draft  the  initial 
standard  could  be  reduced.  This  time  savings,  in  turn,  was  expected  to  reduce 
the  overall  time  period  required  to  produce  a completed  standard  and  get  it 
adopted  by  the  NRC.  The  project  plan  (see  Appendix  B)  outlined  the  activities 
expected  to  occur  as  follows: 

1.  ETIP  prepares  project  plan  to  test  four  changes  from  the  normal 
standard  drafting  process,  for  3 or  4 standards: 

— Hiring  full-time  drafting  subcommittee  chairman 
— Providing  consultant  services  for  data  collection/correlation 
— Providing  clerical/ technical  editing  services 
— Conducting  up  to  14 -day  working  meeting 

2.  ETIP  obligates  funds  for  the  experiment  for  NRC  use 

3.  The  NRC  contracts  with  ANSI  to  conduct  experiments 

4.  ANSI  hires  project  manager  to  coordinate  project  experiments  and 
monitor  effect  of  project  on  development  of  standards 

5.  Each  experiment  differs  in  terms  of  variables  tested,  although 
committees  may  choose  variables  to  be  tested 

6.  ETIP  contracts  with  evaluator  to  conduct  project  evaluation 
to  assess: 

— Whether  standard  development  schedules  were  met 
— Quality  of  standards  produced 

— Whether  a more  ef fective/ef f icient  use  of  working  group 
members  resulted 

— Whether  drafting  time  reduction  offset  project  costs 

7.  ETIP  conducts  further  research  as  needed,  depending  on  results. 


Figure  3 summarizes  the  typical  steps  for  developing  nuclear  standards 
using  the  MSI  voluntary  development  process,  and  illustrates  the  point  in 
that  process  where  the  project  experimental  intervention  occurred. 


FIGURE  3: 


LOCATION  OF  THE  PROJECT  EXPERIMENT 
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The  expectations  underlying  the  experiment  were  most  clearly  outlined 
in  the  project  plan  for  the  proposed  experiment  (see  Appendix  B)  . These 
expectations  are  depicted  in  Figure  4.  The  accuracy  of  these  expectations 
was  reinforced  by  subsequent  interviews  by  the  author  with  ETIP,  NRC,  NBS, 
and  ANSI  staff  members. 
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FIGURE  4:  EXPERIMENTAL  ASSUMPTIONS 


1.  The  major  assumption  underlying  the  voluntary  standards  process  is 
that  industry  production  of  standards  will  produce  greater  likelihood  of 
industry  compliance.  This  assumption  has  not  been  empirically  tested. 
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The  experiments  to  accelerate  the  production  of  first-draft  standards 
by  altering  drafting  conditions  also  were  foreseen  as  only  one  part  of 
a larger  experimental  framework.  As  the  initial  project  plan  discussed, 
this  was  to  be  "the  first  of  a series  of  projects"  in  which  ETIP  planned 
to  "examine  the  relationship  between  voluntary,  or  industry-based  standards 
and  the  regulatory  process."^  Figure  5 illustrates  the  additional  research 
contemplated  by  ETIP  and  discussed  in  the  project  plan  at  the  time  of  the 
initial  experiment,  of  which  this  project  was  only  a first  step. 
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FIGURE  5:  RESEARCH  PROJECTED  IF  THE  PROJECT  PROVED 

SUCCESSFUL  IN  REDUCING  DRAFTING  TIME 


1.  "Accelerating  the  Establishment  of  Standards  and  Their  Adoption  As 
Regulatory  Guides  by  the  Atomic  Energy  Commission,"  Project  Plan,  Washington, 
D.C.:  Experimental  Technology  Incentives  Program,  March  21,  1974. 


IV.  PROJECT  ACTIVITIES 


The  current  research  traced  the  actual  project  activities  from  their 
origins  as  far  as  direct  results  could  be  found.  Figure  6 is  a model  of  the 
project  as  it  occurred,  tracing  the  money  obligated  and  spent,  the  activi- 
ties undertaken,  and  the  resulting  products. 

A detailed  chronology  of  events  and  products  related  to  the  experiment 
was  constructed  by  reviewing  project  documentation  and  by  interviewing  in- 
volved parties  from  the  NRC,  ETIP,  NBS,  and  ANSI.  This  is  reported  as  Ap- 
pendix C.  In  summary,  the  main  events  were  as  follows: 

• 1973-1974  — ETIP,  NBS,  NRC,  and  ANSI  discussed  the  joint 
project  to  accelerate  the  standards  process;  the  project  plan 
and  guidelines  were  prepared. 

• June  1974  — ETIP  obligated  project  funds  to  the  NRC  to  initiate 
the  project  in  accordance  with  the  project  plan  and  guidelines. 

• June  1974  — ANSI  decided  to  contract  out  the  experiment  instead 
of  hiring  a program  manager  as  specified  in  the  plan,  and  amended 
the  interventions  to  be  made. 

• October  1974  — ANSI  contracted  with  Battelle-Northwest  to  coordi 
nate  three  experiments. 

• November  1974  — ANSI  contracted  with  Harlan  Anderson  to  conduct 
a fourth  experiment. 

• March,  1975  — ETIP  developed  a project  plan  for  the  project  eval 
uation . 

• 1975  — Experiments  were  conducted  by  Bat telle  and  Anderson;  no 
satisfactory  contractor  for  the  project  evaluation  was  identified 
Battelle  submitted  its  report  on  the  experimental  process,  with  a 
subjective  assessment  by  the  participants. 

• February  1976  — Anderson  submitted  his  report. 
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e November  1976  — Funds  for  the  project  evaluation  were  deobli- 
gated , and  no  impact  evaluation  was  conducted. 

• 19  76  — A review  of  the  Nuclear  Regulatory  experiment  was  con- 

ducted by  ETIP  staff  and  a case  study  report  submitted  in  January, 
1977. 1 

A.  PROJECT  ORIGINS 

Early  in  1973,  ETIP  was  searching  for  viable  experimental  projects,  par- 
tially through  requests  for  "Qualifications  and  Interests"  Statements.  Dr.  Jim 
Leiss,  Director,  Center  for  Radiation  Research  at  NBS  and  member  of  the  ANSI 
Nuclear  Standards  Management  Board,  proposed  an  experiment  on  accelerating  the 
voluntary  standards  process  in  response  to  the  requests.  ANSI  responded  to 
ETIP's  request  for  "Qualification  and  Interest"  statements  by  suggesting  the 
idea  that  ETIP  reimburse  their  committee  members  for  travel  expenses.  Although 
ETIP  was  not  initially  interested  in  these  proposals,  the  discussions  with  ANSI 
led  to  contact  with  Lester  Rogers,  the  Director  of  Regulatory  Standards  for  NRC. 
Rogers  wanted  to  improve  the  voluntary  standards  process  at  ANSI.  This  interest, 
plus  an  emerging  political  concern  with  reducing  nuclear  plant  licensing  lag, 
eventually  stimulated  ETIP's  interest  in  the  idea.  From  this  time  on,  ETIP 
had  continuing  contact  with  Carl  Johnson  from  the  Regulatory  Standards  office. 

ETIP's  selection  of  the  NRC  and  the  ANSI  volunteer  standards  process  as 
a regulatory  experiment  was  based  on  a strong  public  concern  at  the  time  with 
the  U.S.  energy  problems  and  with  the  costly  delay  in  the  licensing  of  nuclear 
power  plants.  Licensing  delays  were  attributed  in  part  to  the  absence  of 
nuclear  power  plant  standards.  An  experiment  which  would  test  ways  to  speed 
up  the  development  of  standards  was  thus  particularly  attractive. 

1.  Garrity,  Stephen,  "A  Review  of  the  Nuclear  Regulatory  Experiment," 
unpublished  draft,  Washington,  D.C.:  Experimental  Technology  Incentives 

Program,  National  Bureau  of  Standards,  January  11,  1977,  See  Appendix  D. 


B.  PROJECT  PLANNING 


In  March  of  1974,  ETIP  completed  a project  plan  for  the  NRC  experiment. 
(See  Appendix  B.)  The  plan  indicated  that  the  NRC  project  was  the  first 
in  a series  of  projects  examining  the  relationship  between  voluntary  standards 
and  the  regulatory  process.  The  following  four  changes  in  the  subcommittee 
standards-draf ting  process  were  proposed:  paying  a full-time  committee  chair- 

man, using  consulting  services  for  information  gathering,  providing  clerical 
and  editing  assistance,  and  using  more  lengthy  meetings. 

Under  the  plan,  the  Nuclear  Technical  Advisory  Board  (NTAB)  in  ANSI 
(established  by  the  NSMB)  would  assist  the  NRC  in  managing  the  project.  They 
were  supposed  to  select  standards  and  hire  a program  manager.  The  NRC  would 
retain  overall  approval  authority.  While  there  was  no  interagency  agreement 
between  ETIP  and  the  NRC  because  there  was  no  actual  transfer  of  funds,  there 
were  letters  of  commitment  exchanged  between  them.  ETIP  held  an  account 
dedicated  to  the  NRC  project.  ANSI  would  bill  the  NRC  and  the  NRC  would  then 
bill  the  ETIP/NBS  account. 

In  May,  1974,  the  project  plan  was  approved,  and  $100,000  was  obligated 
for  NRC/ ANSI  use. 


C.  PROJECT  PLAN  CHANGES 

Contrary  to  the  project  plan,  ANSI  decided  to  contract  out  the  experiment, 
instead  of  hiring  a program  manager  to  coordinate  the  intervention  and  conduct 
the  experiment.  It  is  unclear  who  at  ANSI  or  the  NRC  made  this  decision  and 
why  it  was  made.  According  to  the  ETIP  review  of  the  project,  "ANSI  ... 
reworded  the  language  of  the  plan  specifying  the  interventions  to  be  used. 
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These  interventions  became  illustrative,  leaving  ANSI  the  option  to  suggest 
other  interventions  it  felt  important ^ 

Thus  in  June,  1974,  ANSI  issued  a request  for  proposals  for  the  exper- 
iment, and  in  August,  Battelle-Northwest  submitted  a proposal  to  look  at 
different  variables  in  three  committees.  (The  standards  chosen  for  the 
experiment  were  selected  partly  because  of  the  urgency  of  need  for  the  par- 
ticular standards.)  A contract  for  $80,000  was  signed  with  Battelle  in 
October,  1974.  Also  in  October,  Harlan  Anderson,  from  Westinghouse-Hanford 
Corporation,  submitted  a proposal  to  study  the  effects  of  one  variable  in  a 
single  committee.  That  proposal  was  funded  in  November  for  $5000. 


D.  EXPERIMENTS  CONDUCTED 

Four  experiments  were  run  using  four  different  committees,  each  working 
on  the  drafting  of  a single  standard.  The  changes  from  the  normal  committee 
operation  included : 

• Use  of  a concentrated  five-day  working  meeting  to  draft  the 
standard  (instead  of  several  one  or  two-day  sessions,  held  over 
many  months) 

• Premeeting  preparation  of  a working  draft  or  outline 

• Provision  of  an  executive  secretary/coordinater 

• Provision  of  clerical/duplicating  support 

• Payment  of  working  group  chairmen 

• Payment  of  travel  and  living  expenses  of  drafting  subcommittee 
members 

• Holding  of  preliminary  organizational  meetings  before  the  five- 
day  work  session 

• Limiting  working  group  membership  to  10  members 


1.  Ibid . , p .16 


20 


Battelle-Northwest  conducted  experiments  with  three  subcommittees,  each 
subcommittee  working  on  a different  standard.  These  included: 


Two  of  the  subcommittees  were  subcommittees  within  the  American  Nuclear 
Society  (ANS)  . One  was  a subcommittee  of  the  Institute  of  Nuclear  Materials 
Management  (INMM)  . (Appendix  E gives  more  detailed  information  on  each  of 
the  experiments  conducted.) 

Each  committee  chair  was  permitted  to  select  the  variables  to  be  tested; 
the  variables  chosen  proved  very  similar.  All  three  committees  tested  the 
five-day  working  meeting,  use  of  an  executive  secretary,  provision  of  cler- 
ical and  duplicating  support,  and  the  limiting  of  the  drafting  subcommittee 
members  to  approximately  ten.  Two  of  the  committees  paid  the  chairmen  and  the 
travel  and  living  expenses  of  the  drafting  subcommittee  members.  Two  subcommit< 
tees  had  subcommittee  members  draft  and  prepare  sections  of  an  initial  working 
draft,  so  that  a working  draft  would  be  ready  for  use  at  the  beginning  of  the 
five-day  work  session.  One  subcommittee  had  the  paid  consultant/ chairman  pre- 
pare a working  outline  for  the  work  session.  And  two  of  the  subcommittees  had 
preliminary  organizational  meetings  before  the  five-day  work  session  was  held. 

Harlan  Anderson  focused  on  only  one  variable.  Anderson  was  chairman  of 
ANSI  Committee  N-ll,  and  used  the  $5000  grant  to  pay  for  the  provision  of  sec- 
retarial  assistance  to  the  very  active  standard s-wri ting  Committee  on  Fuel, 
Control,  and  Moderator  Materials  for  Nuclear  Reactor  Applications  (C-26)  within 
the  American  Society  of  Testing  and  Nuclear  Materials  (AS1M)  . Anderson  was 
also  chair  of  the  experimenting  ASTM  committee,  and  could  directly  observe  the 


Subcommittee 


Standard 


ANS  2.12 


ANS  18.6 
INMM-11 


Guidelines  for  Combining  Natural  Phenomena  and 
Manmade  Hazards  at  Power  Reactor  Sites 
Discharge  of  Thermal  Effluents  Into  Surface  Waters 
Criteria  and  Standards  for  the  Certification  of 
Nuclear  Materials  Managers 


working  sessions  of  the  drafting  subcommittees. 
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E.  REPORTS  PRODUCED 

Battelle  Northwest:  Battelle-Northwest  submitted  a report  on  the  experi- 

ments in  September,  1975.  In  that  report,  Battelle  provided  information  on: 

• The  experiment  conducted,  including  the  subcommittee  members 
and  the  experimental  variables  tested, 

• Costs  of  funding  each  subcommittee's  changes,  and 

• Each  subcommittee's  subjective  ratings,  based  on  a five- point 
scale,  of: 

— the  usefulness  of  the  specific  innovations, 

— the  appropriateness  of  the  subcommittee  composition, 

— skills  of  the  chairman, 

— difficulty  level  of  the  standard, 

— overall  subcommittee  performance. 

Battelle  also  furnished  copies  of  the  draft  standards  produced. 

No  information  was  provided  on: 

• Actual  time  for  producing  the  standards, 

• Standard  quality, 

• Follow-up  on  what  happened  to  the  standards,  once  drafted. 

Harlan  Anderson:  Anderson  submitted  a brief  report  in  February,  1976, 

including  a discussion  of: 

• The  variable  tested, 

• The  background  of  the  consensus  standards  process, 

• His  subjective  assessment  of  results,  based  on  his  own  observa- 
tions and  the  comments  and  reaction  of  other  committee  members, 

• Recommendations  that  further  funding  be  provided, 

• Project  costs. 

Copies  of  meeting  minutes  were  also  provided. 

No  information  was  provided  on: 

• Standards  drafted,  if  any,  or 

• Actual  time  taken  to  produce  standards. 
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F.  PERSONNEL  CHANGES 

A number  of  personnel  changes  occurred  at  the  NRC  and  ETIP  during  the 
course  of  the  project.  In  December,  1974,  Rogers,  the  Director  of  the  NRC's 
Office  of  Regulatory  Standards  and  the  initial  NRC  supporter  of  the  experiment, 
retired.  In  September,  1975,  the  original  ETIP  project  monitor  (Phil  Harter) 
left.  Also,  Victor  Berlin  joined  ETIP  staff,  and  began  discussing  the  need 
for  a wider  focus  on  ANSI's  standards  development  process. 

The  precise  impact  of  these  changes  is  unclear.  They  did  provide  a cer- 
tain discontinuity  with  previous  project  plans  and  implementation.  In  the 
author's  opinion,  they  may  have  lessened  the  understanding  of  and  support  for 
the  project  both  at  the  NRC  and  at  ETIP.  This  was  because  of  the  uncertainties 
created  over  the  value  of  the  original  project  and  the  change  of  interests. 

At  the  minimum,  project  focus  was  somewhat  changed. 


G.  PROJECT  EVALUATION1 

The  initial  plan  for  the  project  foresaw  the  conduct  of  an  evaluation 
of  the  proposed  experiment.  During  March  of  1975,  a project  plan  for  the 
proposed  evaluation  was  prepared  at  ETIP.  The  plan  proposed  to  develop  back- 
ground information  on  the  standards  development  process,  and  to  prepare  a 
detailed  review  of  the  experiment,  including  the  postexperiment  history  of 
the  standard  drafts,  the  project  costs  and  an  identification  of  what 
recommendations  could  be  made. 

The  evaluation  plan  was  approved  in  April,  1975,  and  project  funds  were 
obligated  in  June.  In  December,  ANSI  sent  out  requests  for  proposals  for  the 
conduct  of  the  evaluation,  and  in  January,  1976,  three  proposals  were  submitted. 

1.  The  details  of  this  discussion  were  derived  in  large  part  from  the 
ETIP  Review  of  the  experiment  by  Garrity,  discussed  below.  See  Appendix  D. 
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None  of  the  proposals  was,  however,  accepted  by  the  NRC,  and  in  February, 

ANSI  submitted  a proposal  to  the  NRC  to  conduct  the  evaluation  itself. 

Although  the  NRC  attempted  to  put  through  sole-source  contracting  with 
ANSI  for  the  evaluation,  there  proved  to  be  insufficient  justification  for  a 
sole-source  contract.  Although  ANSI  submitted  a second  and  revised  evaluation 
proposal  in  June,  1976,  no  NRC  action  was  taken. 

H.  SUBSEQUENT  FOLLOW-UP 

In  July,  1976,  Steve  Garrity  joined  the  ETIP  staff.  At  that  time,  he 
was  requested  to  undertake  a review  of  the  nuclear  regulatory  experiment. 

After  considerable  investigation,  the  report  based  on  his  review  was  completed 
in  January,  19  77.  In  this  report,  Garrity  described  and  then  critiqued  the 
experiments  conducted.  A copy  of  the  Garrity  report  is  attached  as  Appen- 
dix D. 

ETIP  requested  that  the  current  assessment  of  the  project  be  conducted 
as  part  of  the  analytic  support  contract  awarded  to  The  Urban  Institute  in 
October  1977.  A major  purpose  of  this  additional  effort  was  to  determine 
if  further  study  was  warranted. 


V.  ASSESSING  THE  ETIP/NRC  PROJECT 


Assessing  the  effectiveness  of  the  nuclear  regulatory  experiment  requires 
comparing  the  expected  project  activities  and  impacts  (best  summarized  in  Figure 
p.  12)  with  the  project  activities  and  impacts  reported  by  the  project  participai 
and  project  documentation  (discussed  in  Section  IV).  Figure  7 summarizes  the 
events  intended  as  contrasted  to  the  events  reported  to  have  occurred. 


A.  PROJECT  ACTIVITIES 


As  the  comparison  indicates,  three  basic  changes  occurred  from  the  expected 
events : 

• ANSI  chose  to  subcontract  the  experiments  rather  than  to  conduct 
the  experiments  through  the  hiring  of  an  in-house  project  manager. 

• More  variables  were  tested  than  were  originally  proposed. 

• The  evaluation  of  project  impacts  was  not  conducted.  As  a 
result,  project  impacts  were  not  ascertained,  and  adequate 
descriptions  of  the  experiments  were  not  provided. 

As  a result  of  the  changes  in  the  project,  the  data  actually  collected  and 


reported  varied  from  what  was  expected  in  the  initial  project  plan 
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INTENDED 

REPORTED 

1. 

ETIP  prepares  project  plan 

Plan  prepared  (1973-74) 

2. 

ETIP  obligates  funds  for  NRC  use 

Funds  obligated  June  1974 

3. 

NRC  contracts  with  ANSI 

NRC  contracts  with  ANSI 

4. 

ANSI  hires  project  manager 
to  coordinate  experiments 
according  to  project  plans 

MSI  contracts  with  Battelle 
(October,  1974)  to  coordinate 
3 experiments 

ANSI  contracts  with  Anderson 
to  conduct  4th  (single-variable) 
experiment  (November,  1974) 

5. 

Experiments  conducted  to 
test  four  variables 

Experiments  conducted;  the  1-8 
variables  tested  overlapped  with 
the  original  4. 

6. 

ETIP  contracts  out  evaluation 
to  assess  impact  of  experiments 

No  evaluation  contract  awar- 
ded. Battelle  submits  report 
on  experimental  process, 
with  subjective  assessment 
by  participants  (1975) . 
Anderson  submits  report 
with  conclusions  but  no 
supportive  data  (1976)  . 

7. 

Draft  standards  are  produced 
more  quickly 

Draft  standards  are  produced 
more  quickly 

8. 

Standards  are  produced  and 
adopted  by  NRC  more  quickly 

Not  addressed  by  experiment 

9. 

Regulatory  lag  decreases, 
operational  safety  increases 

Not  addressed  by  experiment 

10. 

ETIP  conducts  additional  research 
in  area 

ETIP  conducts  follow-up  review 
of  experiment  (1976-1977) 

FIGURE  7:  INTENDED  VS.  REPORTED  EVENTS 


B.  PROJECT  IMPACTS 


Three  criteria  were  identified  in  the  initial  project  plan  as  important 

in  evaluating  the  success  of  the  experimental  project: 

o Time  required  to  produce  the  standard  (as  compared  to  "normal" 
time  requirements)  . 

• Quality  of  the  standard  produced. 

• Cost . 

The  Battelle  and  Anderson  reports  provided  little  information  on  these 
criteria.  However,  the  reports  were  not  intended  from  the  outset  to  be  impact 
evaluations.  What  information  was  provided  in  those  reports  has  been  supple- 
mented to  the  extent  feasible  by  the  follow-up  conducted  as  part  of  the  current 
review. 

1.  TIME  TO  PRODUCE  THE  STANDARD 

I I 

a.  TIME  TO  PRODUCE  INITIAL  STANDARD 

None  of  the  reports  explicitly  indicated  the  exact  length  of  time  required 
to  produce  the  initial  draft  standard.  It  was  possible,  however,  to  reconstruc 

| 

the  time  required  for  the  Battelle  standards. 

Two  of  the  Battelle  subcommittees  produced  a draft  standard  by  the  end  of 
the  scheduled  one-week  working  session.  However,  the  subcommittee  where  no 
preliminary  organizational  meeting  was  held  was  unable  to  produce  a completed 

t 

draft  by  the  end  of  the  working  session  and  subsequent  sessions  were  required 
to  get  agreement  on  the  first  draft.  The  time  required  to  draft  the  three 
Battelle  standards,  from  the  point  of  selecting  the  chairmen  to  completion  of 
the  draft  standard,  was  2+  months,  8 months,  and  9 months  respectively  for  the 
three  standards.'*'  The  Battelle  report  as  well  as  individuals  interviewed 
as  part  of  this  review  concluded  this  time  to  be  faster  than  the  norm  for 


1.  See  footnote  **  on  Figure  8,  p.  34. 
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I production  of  draft  standards,  and  thus  concluded  that  the  experiment  was 
successful  in  terms  of  having  accelerated  the  production  of  the  first  draft 
standard.  Those  NRC,  ANSI,  and  NBS  individuals  interviewed  indicated  that  the 

I time  for  production  of  the  initial  draft  standard  in  the  nuclear  area  typically 
averaged  from  1 to  2 years,  and  sometimes  longer.  One  commented  that  the  ANS 
in  particular  found  it  difficult  to  produce  draft  standards  in  a shorter  time 

] period,  so  that  the  4 to  9-month  time  required  by  the  ANS  subcommittees  to 

j 

produce  the  two  standards  in  the  experiment  was  particularly  distinctive. 

It  was  difficult  to  tell  what  happened  in  the  Anderson  experiment.  The 
report  submitted  was  brief  and  included  no  description  of  the  experiment  con- 
ducted except  that  secretarial  assistance  had  been  provided.  There  was  no 
evidence  that  the  project  traced  the  effect  of  providing  secretarial  assistance 
on  the  production  of  individual  standards.  Nonetheless,  the  Anderson  report 
concluded  that: 

It  was  the  opinion  of  the  officers  and  subcommittee  chairmen 
of  ASTM  C-26  Committee  that  this  experiment  was  very  worthwhile 
and  that  it  helped  to  speed  up  the  nuclear  standard  activities 
of  the  Committee  to  produce  final  national  consensus  standards.  ^ 

b.  TIME  TO  PRODUCE  FINAL  STANDARD 

None  of  the  reports  associated  with  the  experiment  traced  the  time  required 
for  the  draft  standard  to  proceed  through  the  remainder  of  the  standards  develop- 
ment process.  When  this  was  done  as  part  of  the  current  review,  only  one  of  the 
draft  standards  was  found  to  have  survived  the  review  process.  The  time  from 
the  initial  meeting  of  the  drafting  subcommittee  to  the  adoption  of  the  standard 
by  ANSI  was  three  years  and  ten  months.  The  standard  went  to  the  NRC  for  review 
and  adoption  on  August  8,  1978.  The  NRC  did  not  adopt  the  standard,  but  sent 

1.  Anderson,  H.  J.,  "A  Research  Report,  A Single  Variable  Experiment  on 
Accelerating  the  Writing  and  Completing  of  National  Consensus  Standards." 

Final  Report  on  part  of  ETIP  Project  #47  submitted  to  American  National  Standards 
Institute,  Inc.,  New  York,  New  York:  February  27,  19  77,  p.  iii. 
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its  comments  to  the  ANSI  subcommittee  on  November  8,  1978.  The  committee's 
response  has  not  yet  been  transmitted  to  the  NRC.  The  time  required  for  adop- 
tion by  the  NRC  (if  it  is  in  fact  adopted)  remains  unknown. 


c.  WEAKNESS  OF  THE  "TIME"  CRITERION 

The  "length  of  time"  test  is  a difficult  test  to  apply  when  assessing 
the  effectiveness  of  a standard-drafting  experiment.  Factors  other  than 
the  variables  being  tested  may  influence  the  time  required  to  produce  a 
standard  and  get  it  adopted  by  ANSI  or  an  agency,  such  as: 

— the  controversial  nature  of  the  activity  proposed  to  be 
regulated  by  the  standard,  affecting  the  amount  and  length 
of  associated  debate  and  thus  the  length  of  time  required 
to  get  consensus  on  the  standard, 

— whether  the  state  of  knowledge  about  the  activity  is  already 
far  advanced,  lessening  the  time  required  to  produce  the 
draft  standard, 

— political  sensitivity  toward  industry-produced  standards, 
which  may  lead  to  distrust  of  such  standards  and  a tendency 
of  government  personnel  not  to  consider  them, 

— the  experience  and  skill  of  the  particular  interest  group 
in  managing  the  voluntary  standards  process, 

— the  degree  of  prior  experience  the  committee  members  desig- 
nated to  develop  the  standard  had  had  in  working  together 
on  previous  standards, 

— the  urgency  of  need  for  the  standard  (a  factor  of  particular 
import  to  the  experiment  since  ANSI  selected  high  priority 
standards  for  the  project). 

In  addition,  little  data  appears  to  be  available  from  elsewhere  on  the 
"standard"  time  to  develop  a standard.  Since  each  standard  is  unique  and 
typically  developed  by  a separate  group  of  people,  and  since  interest  groups 
follow  somevfaat  different  procedures,  there  is  no  single  standard  development 
process  or  standard  development  time  to  use  for  comparison. 
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2.  STANDARD  QUALITY 

No  precise  criteria  have  been  adopted  for  evaluating  the  quality  of 
standards  produced  through  the  voluntary  standards  process — a problem  for 
anyone  interested  in  evaluating  the  impact  of  attempting  to  accelerate  the 
standards  development  process.  Possible  tests  proposed  in  the  evaluation 
plan  and  by  the  interviewees  included: 

— whether  the  standard  is  endorsed  by  the  relevant  government 
agency,  and  the  number  of  exceptions  made  to  the  standard, 

— whether  agency  staff  find  the  standard  sufficiently  compre- 
hensive and  as  easy  to  use  as  practicable, 

— whether  the  standard  is  actually  adopted  and  used  by  industry, 

— whether  the  standard  is  effective  over  time  in  achieving 
its  goal,  e .g  . , preventing  accidents. 

Tests  proposed  by  those  interviewed  for  evaluating  the  quality  of  the 
first  draft  standard  included: 

— number  of  objections  raised  to  the  standard  when  reviewed 
at  various  stages  of  the  standard  approval  process, 

— • the  length  of  time  required  for  standard  adoption  by  the 
various  organizations  (e.g.,  the  interest  group,  ANSI,  the 
pertinent  agency) . 

None  of  the  reports  associated  with  the  experiment  attempted  to  assess 
the  quality  of  the  standards  produced  using  the  above-mentioned  criteria,  or 
any  others.  The  evaluation  study  which  had  been  projected  to  assess  standard 
quality  and  might  have  possibly  used  these  criteria  was  not  contracted  out. 

a . FOLLOW-UP  FINDINGS 

The  follow-up  review  of  the  standard's  progress  conducted  for  this  report 
included  no  systematic  attempt  to  assess  the  standards  in  terms  of  all  of  the 
above  criteria.  Follow-up  was  conducted  by  1)  discussing  standard  quality  with 
some  of  the  individual  participants,  and  by  2)  tracing  what  happened  to  the  indi- 


vidual standards. 


Opinions  of  interviewees: 


One  NBS  official  indicated  that  those 


individuals  with  whom  he  had  spoken  following  the  experiment  who  had  looked 
at  the  draft  standards  concluded  that  they  were  as  good  drafts  as  those  which 
are  produced  using  the  normal  drafting  process. 

One  NRC  drafting  committee  participant  was  of  similar  opinion.  He  indi- 
cated, however,  that  some  individuals  on  the  drafting  subcommittees  thought 
that  accelerating  the  drafting  stage  was  likely  to  prevent  the  subcommittee 
members  from  getting  as  much  consideration  and  input  from  their  companies, 
leading  to  more  objections  being  raised  at  subsequent  review  stages  and  thus 
lengthening  the  post-draft  review  stage  since  less  time  was  taken  to  get  a 
consensus  at  the  first-draft  stage.  The  NRC  official  disagreed  with  this  con- 
cern, however.  He  thought  that  the  subcommittee  members  got  as  much  input  from 
their  companies  as  through  the  normal  process,  since  they  maintained  constant 
telephone  contact  with  their  home  company  experts  during  the  five-day  working 
session. 

— Tracing  the  progress  of  the  standards  to  adoption:  Only  one  of  the 

standards  (ANS  2.12)  made  it  through  the  entire  standards  development  process. 
The  INMM-11  standard  was  sent  back  to  the  drafting  committee  pending  further 
study.  Reviewers  of  the  standard  at  the  MSI  canvassing  stage  concluded  that 
sufficient  information  was  not  available  on  the  appropriate  criteria  for 
materials  control  to  permit  the  developing  of  standards  for  selecting  materials 
control  managers. 

The  ANS  18.6  standard  was  held  back  by  ANSI  and  never  sent  out  for 
review.  The  particular  standard  involved  environmental  impacts,  and  since 
recent  standards  relating  to  the  environment  had  met  considerable  resistance 
from  environmentalists  and  failed  to  make  it  through  the  ANSI  review  process, 
the  ANSI  review  was  postponed  pending  reevaluation  of  all  the  particular 
ANSI  committee's  standards. 


31 


The  third  standard  (ANS  2.12)  was  successful.  It  was  approved  as  an  ANSI 
American  National  Standard  in  July,  1978,  and  sent  to  the  NRC  for  comment  on 
August  8,  1978.  The  NRC  did  not  adopt  the  standard,  however,  but  returned  its 
comments  to  the  drafting  committee  on  November  8,  1978.  No  further  action  had 
been  reported. 

b.  ASSESSMENT  OF  THE  "QUALITY"  DATA  AND  CRITERION 

It  is  impossible  to  draw  useful  conclusions  from  the  amount  of  data 
collected  in  the  current  follow-up.  It  would  require  a much  more  extensive 
survey  of  project  participants  and  additional  experts,  as  well  as  the  develop- 
ment of  more  precise  standards  for  comparing  standard  quality  to  enable  a 
comparative  assessment  of  standard  quality.  The  development  of  such  criteria 
would  not  be  a simple  task. 

Since  only  one  of  the  standards  made  it  through  the  process  to  the  NRC 
review  stage,  and  factors  other  than  standard  quality  were  responsible  for 
halting  progress  of  two  out  of  three  standards,  it  would  be  impossible  to 
apply  several  of  the  criteria  proposed  for  assessing  standard  quality  (e.g., 
endorsement  by  the  government  agency) . Nonetheless,  the  draft  standards 
evolved  could  be  submitted  for  assessment  to  a group  of  experts  familiar 
with  standards  development.  It  would  also  be  possible  to  submit  a sample 
of  draft  standards  evolved  through  the  regular  drafting  procedure  to  a 
similar  assessment,  to  evolve  comparative  data. 

Similarly,  a survey  could  be  conducted  to  determine  the  percentage  of 
draft  standards  which  normally  survive  the  standards  review  process,  to  see 
if  the  one-third  success  rate  experienced  in  the  experiment  was  typical. 

None  of  these  paths  appear  to  be  desirable  at  this  point  in  time. 
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An  assessment  of  standard  quality  would  be  costly  and  time-consuming  in 
light  of  the  difficulty  in  establishing  objective  criteria  and  the  time  and 
effort  required  to  trace  the  standards  from  the  drafting  stage  through 
ultimate  disposition. 

3.  COST  OF  ACCELERATING  DRAFTING  PROCESS 

The  Battelle  report  indicated  the  costs  of  introducing  the  innovations 
in  each  of  the  three  subcommittess . (Figure  8 illustrates  the  variables  intro- 
duced and  the  costs  per  subcommittee.)  No  comparison  figures  were  either 
available  or  necessary,  since  in  the  typical  drafting  subcommittee  situation, 
all  costs  are  underwritten  by  the  companies  or  governmental  entities  where 
the  participants  work.  The  cost  of  providing  the  secretarial  services  for 
the  Anderson  experiment  was  reported  to  be  $5000,  or  equivalent  to  the  grant 
amount.  However,  it  is  difficult  to  tell  from  either  of  the  reports  submitted 
exactly  how  the  money  was  spent  or  to  allocate  the  costs  among  the  experimental 
variables.  It  is  thus  impossible  to  identify  the  cost  of  implementing  the 
individual  variables. 

No  additional  cost  information  was  gathered  as  part  of  the  current  review. 
If  detailed  records  were  kept  by  the  two  contractors,  however,  such  information 
could  be  retrieved.  It  would  also  be  possible  to  project  such  costs  with  little 
difficulty  for  most  of  the  variables,  e .g . , cost  of  clerical  support. 

4.  SUMMARY  OF  EXPERIMENTAL  CONCLUSIONS 

Authors  of  the  Battelle  report  reported  the  following  changes  as  success- 
ful, based  on  their  observations  and  questionnaire  data: 
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• Use  of  5-day  or  longer  working  session, 

• Use  of  preliminary  organizational  meetings  prior  to  the  working  meeting, 

• Premeeting  preparation  of  an  outline  or  preliminary  draft  by  the 
working  group  members, 

• Provision  of  an  executive  secretary, 

• Provision  of  clerical  and  duplicating  services. 

Payment  of  the  chairman  or  of  the  travel  or  living  expenses  of  the  members  was 
not  deemed  particularly  useful. 

The  Anderson  report  concluded  that  the  provision  of  secretary  services 
was  also  worthwhile  and  speeded  up  the  overall  process  of  getting  standards 
generated  and  accepted. 

In  follow-up  discussions,  one  ANSI  staff  member  reported  that  in  a 
January,  1978  meeting  of  the  planning  committee  of  the  ANSI  Nuclear  Standards 
Management  Board,  attention  focused  on  how  DOE  funding  should  be  spent  to 
accelerate  development  of  fuel  cycle  standards.  The  American  Nuclear  Society 
(ANS)  member  in  attendance  reported  that  ANS  had  drawn  the  following  conclusions 
based  on  ANS'  experience  with  the  nuclear  standards  project: 

• Three  innovations  were  of  particular  usefulness:  provision  of  the 

executive  secretary  and  of  clerical  and  duplicating  support,  and 
use  of  a longer  working  session. 

• Although  in  the  normal  course  of  events,  money  would  not  be  available 
to  support  use  of  these  innovations,  these  innovations  would  be  imple- 
mented in  instances  where  an  agency  or  group  needed  a standard  quickly 
and  was  willing  to  provide  the  funds  needed  to  accelerate  the  standard 
development  process. 

Figure  8 summarizes  the  experimental  variables  tested  in  the  Battelle  and 
Anderson  projects  and  the  results  reported. 
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BATTELLE  EXPERIMENTAL 

ANDERSON 

COMMITTEES 

COMMITTEE 

ANS  2.12  ANS  18.6  INMM-11 

N-ll 

Independent  variables  tested 

xi 

5 -day  working  meeting  * 
Pre-meeting  preparation  of 

X 

X 

Working  outline  * 
Working  draft  * 

X 

X 

X 

Secretary  provided  * 

X 

X 

X 

X 

Clerical/copying  support  * 

X 

X 

X 

Preliminary  meetings 

X 

X 

Paid  chairman 

X 

X 

Paid  member  expenses 

X 

X 

Limited  number 

X 

X 

X 

Dependent  variables  reported 

Time 

8 mos . 

2+  mos.** 

9 mos . 

reported 

Qual i ty 

_2 

shorter 
than  usual 

Cost 

$3825 

$13,925 

$14,770 

$5000 

Participant  opinions  on: 
— usefulness  of 

innovations 

X 

X 

X 

X 

— overall  performance 

X 

X 

X 

X 

* Those  marked  with  an  asterisk  were  reported  effective  in  contributing 
to  the  standards  drafting  process. 

**  A partial  draft  was  completed  within  2+  months,  but  some  sections 
remained  to  be  drafted  and  had  not  been  completed  within  the  following  two 
months.  No  information  is  available  on  the  first  draft  completion  date. 

1.  The  use  of  an  "X"  indicates  that  the  variable  was  tested  in  the 
particular  experiment  conducted  by  that  committee  and/or  findings  reported. 

2.  The  placement  of  an  indicates  that  no  information  was  reported. 


FIGURE  8:  EXPERIMENTS  CONDUCTED  AND  RESULTS  REPORTED 


VI.  MAJOR  FINDINGS  AND  LESSONS  LEARNED 


A number  of  things  were  learned  from  the  experiment.  The  individuals 
reporting  the  experimental  results  reported  that  at  least  some  of  the  experi- 
mental interventions  did  accelerate  the  initial  drafting  process,  particularly 
the  five-day  work  week,  the  provision  of  secretarial  assistance,  and  the 
clerical/duplicating  support.  Subsequently,  an  ANSI  official  interviewed 
indicated  that  several  interest  group  members  had  convened  and  decided  that 
in  instances  where  an  accelerated  process  was  required  and  money  was  available 
to  underwrite  the  costs,  they  had  decided  to  implement  the  three  interventions 
believed  to  be  effective.  On  the  other  hand,  in  the  normal  course  of  events, 
the  costs  of  the  administrative  supports  were  otherwise  deemed  too  expensive 
and  the  money  unavailable  to  sustain  them. 

These  reported  learnings  are  not,  however,  supported  by  the  evidence 
generated  in  the  experiments.  Most  of  the  lessons  learned  from  the  experiment 
are  related  to  the  experimental  process  rather  than  to  the  outcomes  of  the 
experiments  conducted. 


A.  THE  KNOWNS 

An  assessment  of  the  experiments  conducted  indicates  the  following 
weaknesses  inherent  in  the  design  and  conduct  of  the  experiments. 
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1.  The  requirement  that  ETIP  fund  the  project  with  ANSI  by  transfer- 
ring the  funds  through  the  NRC  diminished  the  ETIP  control  over  the 
design,  development  and  implementation  of  the  experiment.  Problems 
also  occurred  in  the  reimbursement  procedures  as  a result  of  the  NRC's 
middleman  status.^ 

2.  There  was  substantial  turnover  of  major  personnel  figures  who  were 
interested  in  the  experiment  as  originally  designed.  This  may  have 
caused  discontinuity  in  the  project  oversight  and  conceptualization, 
and  in  the  project's  political  status  and  backing  (e.g.,  the  dir- 
ector of  the  AEC  resigned;  the  original  ETIP  project  monitor  left)^ 
This  probably  contributed  to  some  of  the  changes  which  occurred  in  the 
project  design  and  implementation,  and  also  led  to  the  problems  in 
implementing  the  evaluation. 

3.  Planning  and  funding  delays  led  to  a permanent  postponement  of  the 
evaluation  planned  to  assess  project  outcomes  and  impacts. 

4.  The  information  provided  by  the  existing  contractors  on  key  project 
variables  and  outcomes  was  incomplete.  The  data  which  were  provided 
were  either  over-analyzed  statistically  in  view  of  the  same  size 
and  lack  of  experimental  controls,  or  totally  unavailable  on  the 
other  hand. ^ 

Even  if  the  anticipated  evaluation  had  been  conducted,  the  results 
probably  would  have  been  disappointing.  This  is  because: 

1.  The  initial  evaluation  planning  appeared  to  be  short-sighted  in 
terms  of  the  scope  of  study  necessary  to  answer  questions  about 
project  impacts.  For  example,  tracing  whether  the  process  acceler- 
ated NRC  adoption  required  an  evaluation  spanning  numerous  years. 

Only  one  of  the  three  standards  had  reached  the  NRC  as  of  August, 

1978.  (See  Appendix  C for  a discussion  of  what  happened  to  the 
individual  standards  developed.)  Major  project  assumptions  could 
not  be  adequately  tested  within  a one  or  even  two-year  evaluation 
period.  In  addition,  there  was  insufficient  up-front  identification 
of  the  measures  to  be  collected. 

2.  The  initial  design  did  not  indicate  how  the  proposed  experiment 
fit  into  major  research  needs,  or  into  the  overall  standards 
development  process.  In  addition,  the  design  did  not  include 


1.  Garrity,  pp.  14-16,  28-29. 

2.  Ibid . , pp.  27-28. 

3.  Ibid.,  pp.  27,  31-38. 
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tasks  to  collect  background  data  on  the  "normal"  development  process. 
Thus,  there  was  no  initial  plan  for  collecting  the  type  of  objective 
comparative  data  needed  to  enable  evaluators  to  assess  whether  the 
experiment  accelerated  the  development  of  nuclear  standards. 

3.  There  was  no  suggestion  of  the  need  to  collect  background  data 

to  examine  the  accuracy  of  the  experimental  assumptions.  Those  who 
proposed  and  undertook  the  experiment  relied  on  the  assumptions  out- 
lined in  Figure  4.  That  is,  the  project  and  evaluation  designers 
presumed  that  by  accelerating  the  establishment  of  standards,  the 
NRC  review  of  proposed  nuclear  plants  would  be  accelerated,  thereby 
accelerating  the  nuclear  plant  licensing  and  approval  process.  No 
background  information  was  proposed  to  be  (or  was  in  fact)  collected 
at  the  outset  about  the  NRC  licensing  and  plant  approval  process 
to  see  what  role  standards  actually  played — to  check  whether  the 
assumptions  about  the  role  of  standards  in  accelerating  plant  li- 
Thus , while  it  is  entirely  possible  that  the  chain  of  assumpltions 
and  expectations  outlined  in  Figure  4 is  correct,  it  is  not  apparent 
from  the  project  documentation  or  interviews  with  interested  parties 
that  these  assumptions  were  reviewed  or  questioned  at  the  outset  of 
the  experiment.  If  they  had  been,  the  questions  about  the  project's 
relevancy  could  have  been  better  addressed  both  during  the  experiment 
and  afterward. 


B.  THE  UNKNOWNS 

Because  of  the  experimental  design  process,  there  remain  many  unknowns 
and  uncertainties  about  the  impacts  of  the  experimental  interventions.  Little 
or  no  information  was  collected  on  such  factors  as: 

• Average  time  for  standard  drafting  and  processing. 

• The  role  of  existing  standards  in  the  plant  licensing  process. 

• Quality  of  the  standards  produced. 

• Cost  of  introducing  the  individual  innovations,  compared  with 
normal  production  costs. 

o The  details  of  the  experiments  conducted. 

It  would  be  difficult  and  costly  to  obtain  information  on  these  factors  at 
this  point  in  the  project. 
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C.  USEFULNESS  OF  FURTHER  STUDY 

Further  study  on  the  impacts  of  the  experiments  does  not  seem  to  be 
justified.  The  finding  that  providing  secretarial  and  clerical  support  and 
longer  working  sessions  accelerate  the  production  of  draft  standards  is  not 
surprising,  and  there  is  sufficient  information  already  available  to  justify 
the  conclusion  that  the  standards  were  drafted  more  quickly  than  was  the 
usual  case  . 

The  issue  which  remains  somewhat  uncertain  is  the  effect  of  accelerating 
the  standard  drafting  process  on  the  consensus  process  that  follows  in  the 
standards  development  process.  This  question  cannot  be  answered  solely  by 
further  review  of  the  experiment  already  completed,  because  the  sample  was 
too  small.  Additional  research  would  be  necessary,  replicating  the  drafting 
stage  experiment  with  a number  of  standards  and  tracing  the  standards  through 
the  remainder  of  the  development  process.  Since  only  one  of  the  standards 
in  the  current  process  proceeded  beyond  the  initial  stages  of  the  review 
process,  further  follow-up  of  the  initial  standards  alone  would  likely  glean 
little  additional  information.  Also,  comparative  information  about  the 
"normal"  drafting  process  is  needed  to  render  the  experimental  data  mean- 
ingful . 

ETIP  thus  has  several  options  relative  to  the  conduct  of  further  study 
on  the  voluntary  standards  process: 

Option  1:  Terminate  the  research  at  the  conclusion  of  the  present 

review. 
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Option  2;  Fund  a limited  research  effort  to  gather  additional  background 
information  about  the  experiments  conducted  and  their  impacts  (e.g.,  the  costs 
of  the  individual  innovations,  the  quality  of  the  standards  produced)  as  well 
as  about  the  ''normal"  development  process  and  time  requirements  in  the  nuclear 
standards  area,  with  which  to  compare  the  experimental  results. 

Option  3:  Fund  a substantial  research  effort  to  identify  the  role  of 

standards,  if  any,  in  accelerating  nuclear  plant  licensing,  using  a research 
design  and  a contracting  process  which  will  address  the  difficulties  experi- 
enced in  the  initial  experiment  and  detailed  in  this  and  the  earlier  review. 
Assuming  that  standards  are  found  to  be  a substantial  factor  in  accelerating 
nuclear  plant  licensing,  then  ETIP  could  consider  the  usefulness  of  investi- 
gating how  and  whether  the  voluntary  standards  process  can  be  successfully 
accelerated  to  develop  those  standards. 

Option  4:  Fund  an  evaluability  assessment  of  the  usefulness  of  conducting 

further  study  of  the  voluntary  standards  process  and  the  impact  of  standards 
of  nuclear  plant  licensing.  This  would  involve  collecting  sufficient  back- 
ground information  about  the  proposed  research  that  the  costs  and  likely 
results  of  the  various  research  options  could  be  assessed  prior  to  commiting 
funds  to  any  particular  research  design. 

ASSESSMENT 

Options  2 and  3 could  be  pursued  if  the  resources  are  available. 

Option  2 would  provide  ETIP  with  additional  impact  information  and  allow  the 
experimental  results  to  be  compared  with  the  data  on  the  "normal"  standards 
development  process.  This  would  provide  a better  context  for  the  experiments 
conducted.  However,  it  is  somewhat  questionable  whether  the  information 


derived  would  be  worth  the  costs. 
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Option  3 would  permit  ETIP  to  address  the  larger  question  of  the  relevancy 
of  attempts  to  accelerate  the  development  of  standards.  In  light  of  the  recent 
3-Mile  Run  nuclear  incident,  the  question  of  the  desirability  of  accelerating 
nuclear  plant  licensing  has  become  particularly  significant,  if  the  accelera- 
tion implies  safety  trade-offs.  An  investigation  of  the  role  of  standards 
in  accelerating  nuclear  plant  licensing  while  concurrently  maintaining  high 
safety  standards  should  therefore  be  seriously  considered.  ETIP's  selection 
of  Option  3 would  require  a substantial  research  commitment  over  a number  of 
years,  however.  This  would  probably  extend  well  beyond  the  scope  of  the 
original  project. 

If  further  research  in  the  standards  area  is  contemplated,  however. 

Option  4 is  the  most  highly  recommended.  The  conduct  of  an  evaluability 
assessment  would  provide  ETIP  with  guidance  as  to  the  areas  where  further 
research  is  desirable  and  likely  to  be  fruitful,  before  significant  resources 
are  committed.  Option  1 is  the  best  selection  at  this  particular  stage  of  the 
ETIP  program,  however,  if  sufficient  resources  cannot  be  devoted  to  pursue 
any  of  the  other  options. 


D.  SUMMARY 

Despite  the  design  and  administration  difficulties  with  the  experiment 
which  was  actually  conducted,  some  useful  information  was  gleaned.  The  drafting 
process  is  believed  to  have  been  accelerated.  However,  the  goals  of  the  experi- 
ment were  not  limited  to  the  standards  drafting  stage. 

To  have  learned  whether  the  acceleration  of  the  drafting  stage  accelerated 
the  entire  development  process  and  NRC  endorsement  would  have  required  beginning 
with  many  more  standards,  tracing  them  for  several  years  to  see  what  happened 
to  them,  and  comparing  their  progress  with  the  norm.  To  have  learned  whether 
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the  availability  of  standards  accelerated  nuclear  plant  licensing  would  have 
required  yet  additional  research.  Considerably  greater  resources  would  have 
been  required  to  conduct  that  additional  research. 

The  current  review  was  useful,  however,  in  highlighting  factors  which 
need  to  be  addressed  by  ETIP  in  the  design  of  future  regulatory  experiments. 

1.  The  process  of  funding  projects  through  an  outside  agency  can 
leave  ETIP  with  less  control  over  the  design,  development  and 
implementation  of  its  experimental  projects.  ETIP  should 
explore  ways  to  ensure  that  the  integrity  of  ETIP  experimental 
designs  can  be  maintained.  Policies  and  mechanisms  for  dealing 
with  derailed  projects  should  likewise  be  developed. 

2.  ETIP  should  assess  proposed  experiments  at  the  project  design 
stage  to  determine  the  probable  validity  of  the  underlying 
experimental  assumptions  and  the  feasibility  of  evaluating 
project  impacts.  Ibis  requires  collecting  background  informa- 
tion about  the  regulatory  process  which  is  the  subject  of  the 
experiment,  the  likelihood  that  the  proposed  changes  can  be 
implemented,  and  the  feasibility  of  collecting  the  data  neces- 
sary to  evaluate  the  project's  ultimate  impacts.  Had  such  a 
preassessment  been  conducted  with  the  current  project,  many  of 
the  problems  associated  with  the  project  design  and  data  col- 
lection might  have  been  foreseen  and  dealt  with.  Much  of  the 
misunderstanding  and  controversy  over  the  value  of  the  experiment 
undertaken  might  have  thus  been  avoided. 
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PROJECT  47  PLAN 

ACCELERATING  THE  ESTABLISHMENT  OF  STANDARDS 
AND  THEIR  ADOPTION  AS  REGULATORY  GUIDES 
BY  THE  ATOMIC  ENERGY  COMMISSION 

I 

BACKGROUND 

This  is  the  first  of  a series  of  projects  in  which  ETIP 
plans  to  examine  the  relationship  between  voluntary,  or  in- 
dustry based  standards  and  the  regulatory  process.  This  re- 
lationship is  varied  and  complex:  It  ranges  from  the  adoption 

by  an  agency  of  voluntary  standards  so  that  they  then  have  the 
force  of  law,  through  the  reference  by  an  agency  to  a standard 
as  an  approved  method  of  doing  something,  on  to  an  independence 
from  the  regulatory  process  except  as  a standard  may  trans- 
gress national  policy  such  as  the  antitrust  laws.  Each  of 
the  varying  relationships  has  its  own  needs  and  concerns.  It 
is  therefore  essential  that  ETIP  experiment  with  these  impor- 
tant relationships  between  voluntary  standards  and  the  regula- 
tory processes  in  an  attempt  to  optimize  them. 

This  project  addresses  the  need  of  the  Atomic  Energy 
Commission  for  the  efficient  and  expeditious  development  of 
standards  which  may  be  used  in  the  licensing  process  for  nu- 
clear reactors.  In  order  for  the  country  to  timely  secure 
the  benefits  of  a developing  technology  and  to  stimulate  in- 
vestment in  it  by  reducing  the  extraordinary  cost  of  delay, 
the  punctual  development  of  hundreds  of  these  standards  is 
essential . 
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A.  The  Need  for  Standards.  The  advent  of  the  recent 
energy  crisis  has  underscored  the  need  to  begin  using  on  a 
widescale  something  other  than  fossil  fuels  for  the  genera- 
tion of  electricity.  While  there  is  considerable  research 
underway  in  an  effort  to  discover  commercially  viable  alter- 
native fuels,  currently  by  far  the  most  promising  is  an  ex- 
panded use  of  nuclear  reactors. 

But  before  a nuclear  power  plant  may  be  used  for  genera- 
ting electricity  (or  any  other  use),  its  design,  construction, 
and  operation  must  meet  the  stringent  standards  which  have  been 
established  by  the  Atomic  Energy  Commission  to  protect  the 
public  health  and  welfare.  Because  the  reactors  are  so  com- 
plex and  because  the  level  of  review  must  be  so  detailed,  this 
licensing  process  is  extremely  time  consuming.  Currently  it 
takes  approximately  ten  years  for  a plant  to  begin  operation 
after  the  decision  was  made  to  build  it;  in  large  part,  this 
is  due  to  the  licensing  process.  This  long  lead  time  means 
that  the  country  must  meanwhile  generate  its  power  from  fos- 
sil fuels  and  hence  has  to  wait  a long  time  to  secure  the 
benefits  of  a needed  technology.  Moreover,  and  perhaps  worse, 
the  long  and  expensive  licensing  process  means  that  a utility 
which  might  be  interested  in  building  a nuclear  plant  must 
wait  a long  time  before  generating  income  on  its  investment, 
which  in  turn  drastically  raises  the  effective  cost  of  the 
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plant  and  forces  a utility  to  tie  up  capital  for  a long  period. 
As  a result,  the  President  has  declared  that  it  is  a national 
objective  to  reduce  this  process  by  several  years. 

One  of  the  main  reasons  for  the  extraordinary  length  of 
time  for  the  ultimate  licensing  is  that  the  Commission  must 
satisfy  itself  that  all  aspects  of  the  plant  --  from  its  loca- 
tion through  its  reactor  design  and  all  support  equipment  -- 
are  safe  and  have  a high  degree  of  freedom  from  faults  and 
errors;  and,  of  course,  the  total  system  must  have  similar 
properties.  If  done  on  a plant-by-plant  basis  the  individual- 
ized assessment  of  each  part,  sub-system  and  total  system  is 
necessarily  time  consuming.  And  it  is  also  subject  to  human 
error  since  at  each  stage  a new  human  judgment  as  to  safety 
and  reliability  must  be  made. 

In  order  to  diminish  the  amount  of  time  needed  to  review 

( 

a proposed  plant,  as  well  as  to  diminish  the  chance  of  error, 
the  Commission  has  encouraged,  supported,  and  given  priority 
consideration  to  activities  leading  to  greater  standardization 
of  nuclear  power  plants  in  terms  of  their  design,  fabrication, 
construction,  testing,  and  operation.  Thus  if  a plant,  or 
part  of  a plant,  meets  a standard,  the  Commission  is  able  to 
approve  that  part  without  a de  novo  and  detailed  examination 
into  all  its  facets  --  from  theoretical  physics  to  industrial 
quality  control. 
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Another  benefit  of  standardization  is  that  as  the  avail- 
able technology  advances  the  standards  can  be  amended  to  in- 
corporate it.  .Thus  they  not  only  diminish  the  amount  of  time 
consumed  by  the  regulatory  process  and  its  chance  for  error, 
they  also  serve  to  advance  the  state  of  technology  actually 
in  use. 

B.  Regulations.  The  Commission  has  established  a series 
of  regulations  which  must  be  met  before  a plant  can  be  licensed 
Some  of  these  regulations  are  general  in  that'  they  state 
basic  policy  goals  of  the  Commission  or  they  are  performance 
criteria  which  must  be  met.  Others  are  quite  specific  and 
detail  precisely  what  must  be  done. 

C.  Regulatory  Guides.  While  the  regulations  must  be 
complied  with,  often  times  there  is  still  considerable  leeway 
as  to  how  that  can  be  accomplished.  For  example,  if  a regula- 
tion requires  that  a part  meet  a certain  performance  level, 
there  may  be  many  ways  of  meeting  that  level  and  many  ways 

of  testing  the  part  to  demonstrate  that  it  does.  Thus,  with- 
out more,  many  of  the  initial  problems  of  an  uncodified  re- 
view process  would  remain:  there  would  still  be  individualized 

review  with  its  consumption  of  time  and  chance  for  error. 

In  order  to  provide  a method  of  standardization  without 
restricting  the  opportunity  for  developing  new  technologies 
which  comply  with  the  performance  requirements,  the  staff  of 
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the  Commission  has  established  a series  of  "regulatory  guides". 
These  guides  inform  applicants  and  licensees  of  ways  accept- 
able to  the  Commission's  regulatory  staff  of  complying  with 
the  mandatory  regulations.  They  implement  the  regulations  and 
are  not  substitutes  for  them.  They  are  not  mandatory,  and  an 
applicant  may  meet  the  regulatory  requirements  in  other  ways 
if  it  can  demonstrate  the  appropriate  compliance.  Thus  the 
guides  are  approved  methods  of  compliance. 

D.  How  the  Regulatory  Guides  are  Established.  The  adop- 
tion of  a regulatory  guide  begins  with  the  development  of  a 
standard  by  either  the  AEC  staff  or  by  the  industry  and  other 
experts  --  suppliers,  users,  theoreticians,  government  person- 
nel and  others  --  through  the  voluntary  standards  program.  As 
it  does  in  other  areas,  the  American  National  Standards  Insti- 
tute (ANSI)  acts  as  a clearinghouse  which  co-ordinates  the 
development  of  standards  within  the  private  sector.  The  Nu- 
clear Technical  Advisory  Board  (NTAB)  of  ANSI  is  responsible 
for  co-ordinating  the  development  of  nuclear  standards,  with 
the  actual  development  of  standards  done  by  committees  estab- 
lished by  professional  societies  and  standards  groups. 

In  order  to  assure  that  the  nuclear  standards  developed 
in  the  ANSI  process  will  be  suitable  for  use  by  the  Commission 
in  its  regulatory  program,  the  Directorate  of  Regulatory 
Standards  co-ordinates  standards,  priorities,  schedules  and 
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standards  writing  activities  with  NTAB.  The  Director  of  Regu- 
latory Standards  is  a member  of  NTAB  and  of  its  Executive  Com- 
mittee. More  than  140  staff  members  of  the  Commission  parti- 
cipate directly  in  standards  writing  groups  of  ANSI,  and  the 
Commission  contributes  more  than  $80,000  annually,  in  funds 
and  services,  to  the  ANSI  efforts.  The  Director  of  Regulation 
has  made  a commitment  that  an  ANSI -approved  nuclear  standard 
will  be  reviewed  by  his  staff  and  action  will  be  initiated  to 
implement  the  standard  within  90  days  or  ANSI  will  be  informed 
as  to  why  the  particular  standard  cannot  be  adopted  as  either 
a regulation  or  regulatory  guide. 

In  1971  the  Commission  requested  ANSI  to  greatly  accelerate 
the  development  of  standards  pertaining  to  nuclear  plants. 

ANSI  responded  by  increasing  its  staff  and  expanding  the  scope 
of  its  activities  to  include  an  intensive  effort  in  establish- 
ing priorities  for  needed  standards,  scheduling  the  development 
of  those  standards,  and  recruiting  standards  writers.  Recent- 
ly over  100  new  working  committees  have  been  formed,  manage- 
ment positions  have  been  filled,  and  a large  number  of  high 
priority  projects  have  been  scheduled. 

The  key  elements  in  producing  quality  standards  which 
may  be  used  by  the  Commission  are  the  efficiency  and  effective- 
ness of  the  standards  writing  groups.  At  present,  the  members 
of  the  individual  standards  wtiting  groups  are  employees  of 
utilities,  architect -eng ineers , equipment  manufacturers,  and 
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of  the  Government,  who  have  expertise  and  experience  in  the 
field  in  which  the  group  is  working.  Their  participation  is 
purely  voluntary  and  they  are  not  separately  paid  for  their 
s tandards -development  work.  For  the  most  part  the  time  spent 
on  developing  standards  is  only  that  which  can  be  spared  from 
their  full-time  duties  with  their  employers  or  in  their  after- 
duty hours.  Because  of  the  need  for  highly  qualified  and 
experienced  persons  on  working  groups,  the  members  are  usually 
key  staff  members  of  their  employers  and  consequently  the 
demands  on  their  time  by  their  employers  is  heavy.  Thus  it 
may  take  a considerable  number  of  weeks,  months,  or  even  years 
in  order  to  secure  enough  of  the  time  of  these  busy  men  to 
fully  develop  a standard. 

The  chairman  of  a committee  has  similar  restrictions  on 
his  time,  and  yet  his  duties  include  the  generation  of  back- 
ground information  which  underlies  a standard  under  develop- 
ment and  the  drafting  and  circulation  of  proposed  standards. 
Since  he  too  has  so  many  demands  on  his  time,  this  process  may 
be  considerably  slower  than  it  would  be  if  the  committee' 
chairman  could  give  the  process  more  of  his  attention. 

The  Commission  is  vitally  concerned  with  developing 
these  standards  as  expeditiously  as  practicable  so  that  the 
country  can  timely  secure  the  full  benefits  of  the  technology 
of  nuclear  power.  Hence  it  is  interested  in  discovering  ways 
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to  accelerate  the  development  of  responsible  standards  for 
use  in  the  regulatory  process. 

D.  What  is  Proposed  Here.  In  this  project  at  least  four 
changes  will  be  made  in  the  normal  operation  of  several  standard 
writing  groups  to  determine  if  any  or  all  of  them  have  a signi- 
ficant effect  on  accelerating  the  establishment  of  responsible 
standards  which  are  approved  by  ANSI  and  adopted  by  the  Com- 
mission. The  four  changes  will  be: 

° hiring  a full-time  working-group  chairman 

° providing  consultant  services  for  col- 

lection and  correlation  of  basic  data 
that  is  required  by  the  group  developing 
the  standard 

0 clerical  and  technical  editing  assistance 

0 bringing  the  group  together  for  up  to  two 

weeks  to  concentrate  fully  on  the  develop- 
ment of  the  standard. 

II 

PURPOSE 

The  purpose  of  this  project  is  to  determine  whether  any 
or  all  of  the  four  changes  in  the  method  of  developing  nuclear 
standards  is  a desirable  method  for  accelerating  the  establish- 
ment of  responsible  standards  which  are  approved  by  ANSI  and 
adopted  by  the  Atomic  Energy  Commission. 
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III 

ANTICIPATED  RESULTS 

It  is  anticipated  that  each  of  the  four  changes  to  be 
made  in  the  usual  method  of  developing  nuclear  standards 
will  accelerate  the  development  of  those  standards  by  enabl- 
ing the  committee  to  more  directly  focus  its  attention  on 
reaching  a consensus.  In  particular,  the  four  changes  are 
expected  to  have  the  following  results: 

A.  Full-time  Chairman.  Since  it  is  the  chairman  who  is 
responsible  for  co-ordinating  the  committee  --  from  the  develop- 
ment of  information  needed  to  reach  consensus  to  the  prepara- 
tion and  circulation  of  draft  standards  --  the  demands  on  his 
time  for  administrative  work  are  considerable.  Yet  it  is  he 
who  must  pass  initial  judgment  on  proposed  standards  and  change 
relevant  comment  on  a proposed  standard  into  a suitable  amend- 
ment to  it;  if  information  is  needed  he  must  know  where  to 
get  it  and,  since  there  is  rarely  a budget  for  such  things, 
he  must  usually  have  to  know  where  he  can  generate  it  for 
free.  Thus  the  chairman  must  necessarily  be  an  expert  in 
his  field  and  must  devote  considerable  time  to  the  perfection 
of  the  standard  on  which  his  committee  is  working.  By  hiring 
a full-time  chairman*  these  duties  could  be  performed  in  a 


* ■ All  the  duties  of  a committee  chairman  would  probably  only 
occupy  half  of  the  time  of  a suitable  expert;  thus  "full- 
time chairman"  means  that  the  chairman  will  be  paid  to  de- 
vote a specified  portion  of  his  professional  time  to  the 
work  of  standards  development. 


10 


much  shorter  period  and  with  considerably  more  concentration, 
which  in  turn  would  mean  the  committee  itself  would  be  ready 
sooner  to  progress  towards  consensus  and  an  acceptable 
standard . 

B.  Consultant  Services.  Many  of  the  high-priority 
standards  require  the  collection  and  correlation  of  consider- 
able amounts  of  information  before  the  committee  can  determine 
whether  there  is  enough  information  on  which  to  reach  a con- 
sensus. Usually  this  information  is  gathered- and  correlated 
by  the  committee  members  themselves,  and  consequently  it  takes 
a considerable  amount  of  time.  If  this  information  can  be 
generated  for  the  committee  by  a consultant  group,  then  the 
time  involved  in  preparing  the  background  information  can  be 
significantly  shortened.  This  too  will  reduce  the  amount  of 
time  required  for  reaching  consensus  by  shortening  the  amount 
of  time  consumed  in  satisfying  their  need  for  information. 

C.  Clerical  and  Editing  Assistance.  As  previously  des- 
cribed, there  is  a considerable  amount  of  administrative  work 
in  the  development  of  a standard.  Clerical  and  editing  as- 
sistance will  be  provided  the  chairman  in  order  to  speed  up 
the  process  of  preparing  initial  and  revised  drafts  of  stand- 
ards and  the  support  documents  which  are  distributed  to  the 
committee.  This  should  facilitate  the  standard -development 
process  by  reducing  the  amount  of  administrative  work  per- 
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formed  by  the  chairman  and  by  providing  professional  help 
experienced  in  the  process  of  codification. 

D.  Bringing  the  Committee  Together.  At  present,  most 
of  the  functioning  of  the  committees  is  by  mail  or  in  iso- 
lated moments  when  its  members  are  together  for  professional 
meetings.  In  this  part  of  the  project,  a committee  will  be 
brought  together  for  as  much  as  two  weeks  to  wota  exclusively 
on  the  development  of  their  standard.  The  free  exchange  of 
ideas  made  available  by  face-to-face  discussion,  and  without 
intervening  duties,  should  mean  that  each  member  of  the 
committee  can  give  his  full  concentration  to  the  development 
of  the  standard  and  any  questions  or  disagreements  which 
arise  can  be  focused  upon  directly  by  the  entire  committee. 

This  should  serve  to  resolve  many  of  the  time  consuming 
questions  and  disputes  which  naturally  arise  during  the  process 

Added  together,  it  is  anticipated  that  these  four  changes 
will  mean  that  responsible  standards  in  very  difficult  areas 
can  be  developed  in  a significantly  shorter  period  than  would 
otherwise  be  the  case. 


IV 

APPROACH 

The  Atomic  Energy  Commission  will  be  the  lead  agency  in 
this  project,  but  because  of  its  regulatory  obligations  it 
will  not  perform  its  duties  any  differently  than  it  currently 
does  with  respect  to  the  development  of  standards  in  the 
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private  sector.  The  overall  management  of  the  project  will 
be  by  the  Nuclear  Technical  Advisory  Board  of  the  American 
National  Standards  Institute,  which  occupies  a .special,  quasi- 
official status  with  respect  to  the  development  of  standards 
-which  are  ultimately  used  by  the  Commission  as  part  of  its 
regulatory  structure. 

The  Nuclear  Technical  Advisory  Board  will  select  three 
(or  possibly  four)  standards  which  are  of  extremely  high 
priority  and  which  are  difficult  to  develop  because  of  a need 
for  a considerable  amount  of  information.  The  areas  selected 
will  also  have  standards  developing  committees  already  in 
existence  and  they  would  be  able  to  be  carefully  monitored 
so  that  the  results  of  this  project  can  be  reliably  measured. 
The  four  areas  are: 

0 site  seismic  design  considerations 

° effects  of  combined  natural  phenomena  - 

design  considerations  (seismic,  flooding, 
tornado,  etc.) 

° fuel  quality  assurance 

0 valve  and  pump  qualification  standards. 

NTAB  will  hire,  on  a part-time  basis,  a program  manager 
to  be  responsible  to  it  for  the  management  of  this  project. 

The  program  manager  will  be  the  chief  administrative  officer 
of  the  project,  and  will  be  responsible  for  hiring  the  respec- 
tive chairmen,  the  clerical  and  editing  assistance,  and  the 
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resoective  consultants;  he  will  be  responsible  for  scheduling  and 
ensuring  that  the  schedules  are  met;  he  will  also  keep  account 
of  how  the  three  or  four  committees  use  their  resources  and 
the  effect  of  this  project  on  the  development  of  the  respective 
standards . 

No  specific  program  for  the  use  of  the  four  changes 
itemized  has  been  determined.  It  is  felt  that  rather  than 
establishing  a set  matrix  for  their  use  before  the  committees 
begin  actual  operation,  it  would  be  best  to  allow  the  commit- 
tees to  draw  upon  these  resources  to  enhance  their  ability 
to  develop  their  respective  standards.  It  will  be  the  duty 
of  the  program  manager  to  keep  detailed  accounts  of  the  use  of 
these  resources  and  to  ensure  that  they  are  used  by  the  various 
committees  in  such  a manner  that  the  effect  of  each  can  be 
appropriately  determined;  that  is,  the  three  or  four  commit- 
tees will  not  duplicate  each  other  in  their  use  of  the  four 
changes.  For  example,  one  (or  more)  of  the  committees  might 
not  have  a full-time  chairman  but  would  extensively  use  con- 
sultants and  editors;  another  might  have  a full-time  chairman 
but  use  consultants  sparingly  because  he  would  develop  the 
information  himself.  If  a committee  feels  some  other  related 
resource  would  greatly  aid  it  in  developing  its  standard,  then 
the.  project  manager  may  approve  the  use  of  the  ^unds  of  this 
project  for  such  other  purpose  upon  the  written  consent  of 
ETIP  and  the  Director  of  Regulatory  Standards  of  the  AEC . 
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It  is  anticipated  that  the  project  will  run  for  approxi- 
mately one  year,  although  the  work  of  some  of  the  committees 
may  extend  somewhat  beyond  then.  The  approximate  cost  is 
anticipated  to  be  broken  down  as  follows: 

$20,000  - Program  manager  (part - t ime) 

55.000  - 3 chairmen  (each  with  technical  expertise 

in  his  field;  each  hired  on  1/4  - 1/2  time 
.basis)  plus  clerical  and  editing  assistance 

25.000  - Consultant  services;  plus  1 two-week  meeting 

$100,000 

It  is  anticipated  that  the  chairmen  will  be  individuals 
who  have  the  relevant  technical  expertise  and  who  are  current- 
ly employed  on  a full-time  basis  by  a firm  interested  in  the 
prompt  licensing  of  nuclear  power  plants;  the  hope  is  that 
an  employer  will  permit  a potential  chairman  to  be  hired  on 
a part-time  basis  for  this  standard's  development  work  because 
of  its  importance.  Another  source  for  such  chairmen  are  in- 
dividuals who  have  recently  retired  from  such  an  active  occupa- 
tion . 

V 

EVALUATION 

The  final  evaluation  of  this  project  is  whether  the  ap- 
proaches to  standards  development  which  are  undertaken  are  a 
desirable  method  of  accelerating  the  establishment  of  respons- 
ible standards  which  a re  adopted  by  the  AEC  and  which  hasten 
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the  licensing  of  power -generating  reactors.  In  large  measure, 
this  evaluation  will  be  based  on  the  subjective  opinion  of 
those  who  are  familiar  with  the  standards  development  process 
by  comparing  what  occurred  in  this  project  with  what  they 
would  have  anticipated  would  have  been  the  case  under  normal 
circumstances.  Thus  the  evaluation  will  be  based  on  the  fol- 
lowing questions: 

0 Have  the  schedules  for  the  standard  development 

(both  the  overall  schedule  already  .determined , 
and  the  particular  schedule  determined  by  the 
project  manager  for  this  project)  been  met? 

0 Are  the  standards  which  result  from  the  project 

of  high-quality,  in  that  they  are  readily  ac- 
ceptable by  the  Commission  and  in  the  opinion 
of  the  Commission’s  staff  they  are  comprehensive 
and  as  easy  to  use  as  is  practicable. 

0 Did  the  project  generate  a more  effective  and 

efficient  use  of  the  working  group  members? 

° Was  the  time  from  initiation  to  adoption  by 

the  Commission  reduced  sufficiently  to  offset 
the  cost  of  the  project? 

If  it  is  determined  that  aspects  of  this  project  have 
significantly  reduced  the  amount  of  time  necessary  for  the 
development  and  adoption  of  standards,  an  inquiry  will  be 
made  as  to  how  much  the  time  needed  to  secure  ultimatp  licensing 


16 


can  be  reduced  by  the  adoption  of  a comprehensive  code  of 
regulations  and  regulatory  guides.  If  that  period  can  be 
significantly  reduced  by  such  a code,  then  an  inquiry  will  be 
made  as  to  why  the  private  sector,  which  would  profit  financial 
ly  from  the  reduction,  has  not  undertaken  similar  efforts 
in  the  past.  An  inquiry  will  also  examine  what  may  be  done 
to  improve  the  existing  system  and  to  better  use  the  resources 
of  the  private  sector. 

Another  inquiry  will  be  made  as  to  the  public  benefits, 
such  as  improved  public  safety  or  reduced  manpower  needs, 
which  stem  from  the  adoption  of  such  codes,  and  whether  the 
Government  should  routinely  sponsor  similar  efforts. 

The  evaluation  will  be  conducted  by  a contractor  hired 


for  the  purpose  by  ETIP. 
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APPENDIX  C 
DETAILED  CHRONOLOGY 


Early  1973 

1.  ETIP  issues  a request  for  "Qualification  and  Interest"  statement. 

ETIP  placed  a notice  in  the  Commerce  Business  Daily  which  broadly  described 
ETIP's  mission  and  requested  Q and  I statements  as  well  as  suggestions  for 
work  in  the  regulatory  area.  This  request  was  to  be  the  first  of  a two-stage 
procurement . 

Early  1973 

2.  ANSI  response.  The  American  National  Standards  Institute  (ANSI) 
responded  to  the  request.  They  suggested  the  idea  that  ETIP  reimburse  their 
committee  members  for  travel  expenses.  According  to  Garrity,  "the  idea 
didn't  impress  people"  at  the  time. 

Early  1973 

3.  Second  stage  of  procurement  dropped.  According  to  Garrity,  the 
first  phase  served  to  bring  ETIP  in  contact  with  some  qualified  firms.  The 
second  stage  "died  a natural  death." 

1973 

4.  Contact  with  Rogers.  The  discussions  with  ANSI  led  to  contact 
with  Lester  Rogers,  the  Director  of  Regulatory  Standards  for  AEC.  He  was 
interested  in  improving  the  voluntary  standards  at  ANSI.  From  this  time  on, 
ETIP  had  continuing  contact  with  Johnson  who  worked  for  Rodgers. 

March  21,  1974 

5.  ETIP  completes  a project  plan  for  the  NRC  experiment.  The  plan 

indicated  that  the  NRC  project  was  the  first  in  a series  of  projects  examining 
the  relationship  between  voluntary  standards  and  the  regulatory  process.  The 
plan  lacked  a broad  perspective  and  did  not  relate  the  "experiment"  to  overall 
standards  development  or  to  industry.  The  following  four  interventions  were 
proposed:  paying  a full-time  committee  chairman,  using  consulting  services 

for  information  gathering,  clerical  editing  assistance,  and  more  lengthy 
meetings . 

According  to  the  plan,  the  Nuclear  Technical  Advisory  Board  of  ANSI 
would  manage  the  project,  select  standards  and  hire  a program  manager.  * 

AEC  would  retain  overall  approval  authority. 

Under  the  plan,  the  Nuclear  Technical  Advisory  Board  (in  ANSI) 
would  manage  the  project.  They  were  supposed  to  select  standards  and  hire 
a program  manager. 

6.  No  Interagency  Agreement.  There  was  no  interagency  agreement  because 
there  was  no  actual  transfer  of  funds.  By  congressional  mandate,  AEC  could 
only  receive  funds  appropriated  by  congress.  As  a result,  ETIP  held  an 
account  dedicated  to  AEC.  ANSI  would  bill  AEC  and  AEC  would  then  bill  the 

NBS  account. 

April  12,  1974 

7.  ETIP  sends  project  plan  to  AEC. 
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May  23,  1974 

8.  The  assistant  secretary  for  science  and  technology  approves 
the  plan. 


June  12,  1974 

9.  ETIP  obligates  $100,000  for  AEC/ANSI  use.  According  to  the  case 
study,  ETIP  lost  effective  control  of  the  project  at  this  time. 

June  1974 

10.  ANSI  activity  not  in  conformance  with  several  plan  provisions. 

Instead  of  hiring  a program  manager  to  coordinate  the  intervention  and 
conduct  the  experiment,  ANSI  decided  to  contract  out  the  experiment,  con- 
trary to  the  plan.  It  is  not  known,  who  at  ANSI/NRC  made  these  decisions. 
According  to  Garrity's  case  study,  "ANSI  . . . reworded  the  language  of 
the  plan  specifying  the  interventions  to  be  used.  These  interventions 
became  illustrative,  leaving  ANSI  the  option  to  suggest  other  interven- 
tions it  felt  important."  (p.  16) 

June  1974 

11.  ANSI  issues  a request  for  proposal  for  the  experiment. 

August  28,  1974 

12.  Battelle-Northwest  submits  a proposal  to  ANSI.  The  proposal 
said  Battelle  would  look  at  different  variables  in  three  committees. 

September  1974 

13.  Two  committees  meet.  Two  of  the  four  committees  to  be  involved 
in  the  experiment  hold  their  first  meeting. 

October  15,  1974 

14.  Contract  is  signed  with  Battelle.  The  contract  was  funded  for 
$80,000. 

October  29,  1974 

15.  Harlan  Anderson  submits  a proposal.  Anderson,  from  Westinghouse- 
Hanford  Corporation,  submitted  a proposal  to  study  the  effects  of  one  vari- 
able in  a single  committee. 

November  7,  1974 

16.  ANSI  informs  Anderson  of  proposal  acceptance.  The  contract  was 
funded  at  $5,000. 

December  31,  1974 

17 . Rogers,  Director  of  AEC^s  Office  of  Regulatory  Standards,  retires. 
January  19,  1975 

18.  The  AEC  becomes  the  Nuclear  Regulatory  Commission  by  executive  order. 
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March  25,  1975 

19 . Development  of  a project  plan  for  the  evaluation  of  the  experiment. 
According  to  Garrity's  case  study,  "The  plan  considered  backgrounding  of 
standards  development  process,  a detailed  review  of  the  experiment,  the  post- 
experiment history  of  the  drafts,  the  project  costs  and  what  recommendations 
could  be  made."  One  problem  with  the  plan  was  the  requirement  that  substan- 
tial process  data  be  collected,  even  though  the  experiment  was  half  over. 

April  18,  1975 

20.  Plan  approval.  The  assistant  secretary  for  science  and  technology, 

Department  of  Commerce,  approved  the  project  plan  the  evaluation  of  the 

experiment . 

June  16,  1975 

21.  ETIP  funds  for  the  evaluation  are  obligated  to  NRC . The  amount 
set  aside  for  the  evaluation  was  $15,000. 

September,  1975 

22.  Berlin  joins  ETIP.  Early  in  his  tenure  he  became- involved  in  the 
project,  arguing  for  a wider  focus  on  ANSI's  standards  development  process. 

September  1975 

23.  Battelle  submits  its  final  report  dealing  with  the  three-committee 
experiment . 

September  1975 

24.  Harter  leaves  ETIP. 

December  2,  1975 

25.  NRC  asks  ANSI  to  prepare  a proposal  for  the  evaluation. 

January  1976 

26.  ANSI  becomes  dissatisfied  with  reimbursement  procedure.  ANSI 
billed  NRC  which  in  turn  billed  NBS.  However,  NRC  held  the  bills  until 
a large  enough  amount  was  accumulated.  According  to  the  case  study, 

ANSI  needed  faster  processing  of  bills  and  also  wanted  reimbursement  of 
precontract  costs. 

Late  January  1976 

27 . ANSI  sends  out  requests  for  proposal  for  conduct  of  the  evaluation. 
ETIP  was  unhappy  with  the  RFP. 

January  30,  1976 

28.  Three  proposals  submitted  to  ANSI.  The  bidders  were  Stafco, 

Rockwell  International,  and  Nuclear  Services  Corporation.  All  three  proposals 
were  unacceptable  to  ETIP. 
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February  11,  1976 

29.  ANSI  sent  its  own  proposal  to  NRC  to  do  the  evaluation.  (Thus,  ANSI 
was  in  the  unusual  position  of  bidding  to  do  a job  and  issuing  RFPs  to  firms 
to  do  the  same  job.) 

February  27,  1976 

30.  Anderson  submits  his  final  report  to  ANSI. 

Spring  1976 

31.  NRC  attempted  to  put  through  sole-source  contracting  with  ANSI. 

The  Garrity  case  study  says,  the  attempt  "was  running  into  delays  because 

NRC  was  uncertain  that  ANSI  was  uniquely  qualified  to  conduct  the  evaluation." 
(43) 

June  14,  1976 

32.  ANSI  sent  a second,  revised  proposal  for  evaluation  to  NRC. 

However,  NRC  failed  to  act  on  the  second  proposal  either.  A year  passed 
since  ETIP  had  obligated  the  evaluation  contract  funds. 

July  21 , 1976 

33.  NRC  asks  ETIP  to  take  control  of  the  evaluation.  According  to 
Garrity,  Rivenbark  and  Johnson  from  NRC's  Office  of  Standards  Development, 
were  simply  not  interested  in  the  evaluation,  partly  because  NRC  was  not 
directly  involved  in  the  experiment  itself.  Also,  Victor  Berlin  wanted  to 
expand  the  scope  of  the  evaluation  to  include  NRC  processes.  According  to 
Garrity,  "This  may  have  turned  them  off."  In  his  case  study,  Garrity 
describes  NRC  as  reluctant  to  have  any  one  study  their  processes.  Finally, 
ANSI  was  annoyed  at  the  rate  of  reimbursement  and  NRC  was  reluctant  to 
continue  its  accounting  function. 

November  29,  1976 

34.  ETIP  deobligates  funds  for  the  evaluation  of  the  experiment. 
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1.0  Introduction 


The  Experimental  Technology  Incentives  Program  (ETIP) 
conducted  an  experiment  with  the  Nuclear  Regulatory  Commission 
(NRC)  from  1974  to  1976.*  The  experiment  consisted  of  two 
phases:  one  which  involved  changing  experimentally  some 

parts  of  the  process  used  to  develop  nuclear  regulatory  standards, 
and  another  which  was  supposed  to  evaluate  the  effects  of  these 
changes.  The  purpose  of  this  paper  is  to  review  the  NRC  ex- 
periement  and  discuss  the  implications  for  the  conduct  of  future 
experiments . 


Approach  to  the  Review 

The  approach  used  in  developing  this  paper  was  to  review 
activities  and  documents  generated  over  the  life  of  project 
by  the  groups  involved.  The  majority  of  the  documents  reviewed 
are  included  as  references  in  this  paper.  The  major  events 
of  the  project  are  listed  chronologically  in  the  appendix. 


When  the  majority  of  the  project  literature  and  information 
was  examined,  it  was  decided  to  divide  the  paper  into  several 


*It  should  be  noted  that  during  the  time  period  of  this  experiment 
the  Atomic  Energy  Commission  (AEC)  was  abolished  and  the  Nuclear 
Regulatory  Commission  was  established.  This  paper  will  include 
both  names,  even  though  for  purposes  of  the  experiment  the 
two  organizations  were  essentially  the  same. 
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parts  which  could  be  reviewed  and  analysed  independently  of 
other  parts.  The  resulting  sections  represent  various  time 
segments  of  the  project,  with  some  sections  having  more  structured 
examinations  than  others.  There  are  sections  on  ETIP  and  NRC 
background,  initiation,  design,  implementation,  analysis,  and 
evaluation.  A final  section  presents  conclusions  and  recommenda- 
tions for  the  NRC  experiment  and  ETIP  experiments  in  general. 
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2.0  Background 

2.1  The  ETIP  Program 

The  Experimental  Technology  Incentives  Program  (ETIP)  seeks 
to  affect  public  policy  and  process  to  facilitate  technological 
change  in  the  private  sector.  ETIP's  goal  is  the  development 
of  a set  of  policy  guidelines  on  technological  change  and  the 
body  of  knowledge  necessary  for  their  effective  use.  This 
goal  is  being  pursued  through  the  examination  and  experimenta- 
tion with  government  policies  or  practices  in  regulatory, 
economic  assistance,  procurement,  and  research  and  development 
areas.  ETIP  works  with  government  agencies  and  provides  assist- 
ance and  funding  for  the  conduct  of  studies  and  experiments 
with  policies. 

» 

The  regulatory  policy  area  at  ETIP  is  interested  in  exam- 
ing  the  relationship  between  government  regulation  and  the 
rate  or  direction  of  technological  change  in  the  private 
sector.  One  focus  is  on  the  regulatory  development  process 
which  often  involves  excessive  time  delays.  This  lag  problem 
is  exemplified  in  the  nuclear  regulatory  arena  where  standards 
are  used  by  the  Nuclear  Regulatory  Commission  in  its  liscensing 
process  for  power  plants. 


2.2  The  Nuclear  Regulatory  Commission 

The  Nuclear  Regulatory  Commission  (NRC)  is  an  independent 
government  agency  concerned  with  the  civilian  uses  of  nuclear 
energy.  Its  purpose  is  to  "assure  that  the  civilian  uses 
of  nuclear  materials  and  facilities  are  conducted  in  a manner 
consistent  with  the  public  health  and  safety,  environmental 
quality,  national  security,  and  the  antitrust  laws"  (Office 
of  the  Federal  Register,  1976,  p.588). 

The  Commission  fulfills  its  purpose  through  the  licensing 
and  regulation  of  nuclear  materials  and  facilities.  The  major 
program  components  of  the  commission  are  the  Offices  of  Nuclear 
Reactor  Regulation,  Nuclear  Material  Safety  and  Safeguards, 
Nuclear  Regulatory  Research,  Standards  Development,  and  Inspec- 

. I 

tion  and  Enforcement.  Some  examples  of  commission  activities 

» 

are  the  (Office  of  Federal  Register,  1976,  p.  588): 

1.  Licensing  of  nuclear  power  plant  construction  and 
operation . 

2.  Licensing  the  use,  processing,  and  transport  o£ 
nuclear  materials, 

3.  Development  and  implementation  of  rules  and  regula- 
tions governing  licensed  nuclear  activities. 
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4.  Conduct  of  public  hearings  on  radiological  safety, 
environmental,  and  antitrust  matters,  and 

5.  The  development  of  effective  relationships  with  the 
states  regarding  the  regulation  of  nuclear  material. 

The  major  focus  of  the  commission  is  on  the  use  of  nuclear 
energy  to  generate  electric  power  (Office  of  the  Federal 
Register,  1976  , p588).  Before  a nuclear  plant  can  be  used  to 
generate  electricity,  NRC  must  approve  the  design,  construc- 
tion, and  operation  of  the  plant  in  accordance  with  the 
Commission's  established  set  of  regulations.  This  can  be  a 
lengthy  review  due  to  the  complexity  of  power  plants  and  the 
need  to  consider  individual  designs  and  subsystems.  In  1974, 
it  was  estimated  that  it  took  approximately  ten  years  from  the 
decision  to  build  a plant  to  the  beginning  of  its  operation, 
and  a major  part  of  this  period  was  devoted  to  the  approval 
process  (ETIP,  1974,  p.2). 

One  approach  used  by  the  Commission  to  reduce  the  time 
required  is  that  of  encouraging  standardization  in  nuclear 
facilities.  The  Commission  issues  regulatory  guides  offering 
approved  methods  of  compliance  with  the  mandatory  regulations 
(ETIP,  1974,  pp.  4-5). 
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Regulatory  guides  are  developed  either  by  the  commission 
or  by  experts  in  industry,  academia,  or  government  through  a 
voluntary  standards  program  (ETIP,  1974,  p.5).  This  program 
is  coordinated  within  the  American  National  Standards  Institute 
(ANSI)  by  the  Nuclear  Technical  Advisory  Board  (NTAB) . The 
Board  establishes  and  manages  committees  of  experts  for  de- 
veloping nuclear  standards.  These  standards  are  submitted 
to  NRC  for  adoption.  In  order  to  coordinate  NTAB  output  with 
NRC  needs  and  ensure  suitability  of  standards,  the  director 
of  the  Office  of  Standards  Development  at  NRC  participates  as 
a member  of  NTAB  and  its  executive  committee  (ETIP,  1974,  p.6). 

2 . 3 Development  of  the  Experiment 

The  new  energy  conscious  climate  of  the  early  1970's 
focused  national  attention  on  methods  for  producing  electricity 
(ETIP,  1974,  pp.  2-3).  With  nuclear  reactors  being  one  of 
the  more  attractive  alternatives  to  fossil  fuels,  interest 

1 

in  building  more  nuclear  plants  increased.  However,  it  was 
taking  a long  period  of  time  to  bring  plants  on-line,  in  part, 
due  to  the  lengthy  licensing  process  of  AEC.  In  1971  the 
Commission  requested  that  ANSI  accelerate  their  voluntary 
standards  program  to  help  reduce  these  time  delays  (ETIP  1974, 
p.6).  ANSI  increased  its  efforts,  but  even  in  1973  it  was 
still  the  Commission's  view  that  more  improvements  in  the 
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process  were  needed  (Rodgers,  1973). 

ETIP's  involvement  with  this  problem  began  in  early 
1973.  ETIP  had  issued  an  announcement  in  the  Commerce 
Business  Daily  presenting  ETIP  interests  and  requesting  opinions 
from  private  firms  (Harter,  Note  1).  This  was  the  first  stage 
of  a two-step  solicitation  for  developing  some  contracts  with 
qualified  firms  (Van  Wyk,  Note  2) . ANSI  responded  to  this 
request  by  proposing  an  experiment  with  the  voluntary  standards 
process.  In  an  attempt  to  accelerate  the  work  of  their  standards 
committees,  ANSI  proposed  to  reimburse  committee  members  for 
travel  expenses  to  meetings.  While  initial  ETIP  interest  was 
low,  further  discussion  of  the  idea  with  two  members  of  NTAB 
indicated  that  there  might  be  more  experimental  potential  than 
originally  proposed  (Harter,  Note  1) . 

These  initial  discussions  led  to  contact  with  another 
NTAB  member,  Lester  Rodgers,  the  Director  of  Regulatory 
Standards  for  AEC.  Rodgers  was  very  interested  in  improving 
the  voluntary  standards  development  process  at  ANSI 
(Rodgers,  1973).  He  provided  the  contact  ETIP  needed  to* make 
the  project  relevant  to  a government  policy  maker  and  to  fulfill 
the  ETIP  model  of  working  through  other  government  agencies. 


Development  of  a formal  plan  for  an  experiment  began  in 
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early  1974  with  discussions  between  AEC  and  ETIP . The  focus 
remained  voluntary  standards  committees  of  ANSI,  even  though 
AEC  itself  also  had  a part  in  the  overall  standards  process. 

These  committees  depended  on  volunteer  experts  from 
government,  academia,  and  industry.  They  lacked  sufficient 
support  services  to  help  construct,  produce,  and  distribute 
drafts.  Both  ETIP  and  AEC  were  interested  in  learning  if 
alteration  of  these  conditions  might  help  reduce  the  time  needed 
to  draft  a standard. 
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3 . 0 The  NRC  Experiment 

/ 

3.1  Initiation  of  the  Experiment 

The  general  approach  in  initiating  an  experiment  from  the 
ETIP  perspective  is  to  first  make  contact  with  another  govern- 
ment agency  and  discuss  the  feasibility  of  doing  an  experiment. 
If  the  discussions  indicate  some  potential  for  collaboration, 
work  is  started  on  a project  plan  and  other  related  documents, 
which  must  be  approved  by  both  ETIP  and  the  agency  for  a project 
to  begin.  In  the  NRC  experiment,  the  project  plan  was  developed 
primarily  at  ETIP  and  was  used  in  the  approval  process  in  the 
Department  of  Commerce  to  justify  a transfer  of  funds  to  AEC. 

AEC  then  used  the  plan  to  contract  the  experimental  work  to 
ANSI  while  AEC  maintained  overall  approval  in  the  project. 

3.1.1  Planning 
Describing  the  Problem 


The  project  plan  provided  an  overview  of  the  voluntary 
standards  process  including  the  need  for  standards,  their  de 
velopment  and  use,  and  how  the  various  groups  (ANSI,  AEC 
and  industry)  were  involved.  The  main  problem  focus  was  on 
the  ANSI  part  of  the  process  and  hoV  it  might  be  accelerated 
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through  alteration  of  conditions  under  which  drafting  committees 
worked.  While  the  plan  was  sufficient  for  an  overall  perspec- 
tive, several  issues  should  be  noted: 


1.  The  relevant  background  and  interests  of  the  groups 
involved  with  the  experiment  were  not  all  discussed  in  enough 


first  in  a series  of  projects  examining  the  relationship  be 
tween  voluntary  standards  and  the  regulatory  process  (ETIP, 
1974,  p.l) . Documents  available  to  this  writer  did  not  in- 
dicate an  overall  strategy  involving  a series  of  projects, 
and  thus  the  ETIP  objectives  for  the  project,  both  in  sub- 
stantive and  methodolgical  areas,  were  unclear. 


enough  for  one  to  understand  the  industry  stake  in  the  voluntary 
standards  program  or  its  acceleration.  It  was  indicated  that 
industry  suffered  a long  delay  to  eventual  plant  operation, 
•(ETIP,  1974,  p.2) . It  is  unclear  how  changes  in  the  ANSI 
standards  program,  which  provides  input  to  NRC  and  involves 
industry,  would  have  an  impact  on  industry  in  the  long  run, 
particularly  in  terms  of  technological  innovation,  a key 
ETIP  interest. 


detail . 


•For  ETIP  the  plan  indicated  that  work  with  NRC  was 


•The  perspectives  of  industry  were  not  discussed 
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•The  project  plan  discussed  some  of  the  standards  de- 
velopment process  at  ANSI  but  avoided  an  overall  perspective 
of  ANSI  functions  and  problems.  Thus,  while  the  focus  was  on 
important  administrative  problems  of  Committees,  equally  im- 
portant technical  issues  in  standards  development  were  not 
discussed  (Swisher,  Note  3) . Also,  the  other  ANSI  functions 
in  the  process,  such  as  coordinating  reviews  of  standards  in 
the  field,  were  not  discussed. 

•The  standards  process  at  NRC  itself  was  only  brief- 
ly discussed. 

2.  While  it  is  unreasonable  to  expect  that  extensive 
discussions  on  each  group  would  appear  in  the  plan,  the  general 
indication  from  these  shortcomings  is  a lack  of  overall  per- 
spective of  the  problem.  There  is  no  way  to  assess  the  overall 
importance  of  the  process  at  ANSI,  and  potential  subsequent 
impacts  due  to  its  acceleration,  against  the  rest  of  the  process 
at  NRC  and  in  industrial  application.  This  would  create 
problems  later  during  implementation  and  evaluation  phases  of 
the  experiment. 

Proposing  Some  Changes 

With  the  problem  focus  on  the  time  delay  and  working 
conditions  of  committees  at  ANSI,  an  experiment  in  altering 
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some  of  the  working  conditions  was  proposed.  The  purpose  of 
the  project  was  "to  determine  whether  any  of  all  of  the  four 
changes  in  the  methods  of  developing  nuclear  standards  is 
a desirable  method  for  accelerating  the  establishment  of  re- 
sponsible standards  which  are  approved  by  ANSI  and  adopted 
by  the  Atomic  Energy  Commission"  (ETIP,  1974,  p.8) . The 
four  changes  suggested  were  paying  for  a full  time  committee 
chairman,  utilization  of  consulting  services  for  information 
gathering,  clerical  and  editing  assistance,  and  bringing  the 
committee  together  for  more  lengthy  meetings. 

Approach  to  Conducting  the  Experiment 

The  experiment  involved  several  agencies  and  some  steps 
were  proposed  in  how  the  project  was  to  be  conducted.  These 
were  as  follows : 

1.  ETIP,  after  receiving  approval  of  the  project  plan 
within  the  Department  of  Commerce,  would  transfer  funds  to 
AEC1. 

2.  AEC  would  contract  with  ANSI,  via  a sole  source  pro- 
cedure, to  conduct  the  experiment  for  a period  of  one  year. 

^■It  should  be  noted  that  AEC  was  only  allowed  to  use  appropriated 
funds.  Thus,  it  could  not  accept  funds  from  ETIP.  The  actual 
arrangement  was  to  establish  an  account  at  MBS  dedicated  to 
the  AEC  project.  AEC  would  bill  this  account. 

(Penn,  Note  7) . 
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AEC  would  still  retain  overall  approval  authority. 

3.  The  Nuclear  Technical  Advisory  Board  (NTAB)  of  ANSI 
would  be  responsible  for  the  overall  management  of  the  project. 
This  would  include  selecting  the  standards  to  be  used  in  the 
experiment  and  hiring  a program  manager  to  run  the  project. 

4.  The  program  manager  would  hire  chairmen  for  the  test 
committees , hire  and  coordinate  support  services , and  collect 
data. 

5 . The  individual  committees  to  participate  in  the  ex- 
periment would  have  the  options  to  select  interventions  as 
needed  in  their  particular  situation,  even  to  the  extent 

of  choosing  interventions  not  in  the  project  plan.  The  program 
manager  would  in  a 11  cases  be  responsible  for  approving  inter- 
ventions and  assuring  that  their  impacts  could  be  assessed 
later . 

The  most  notable  factor  about  these  arrangements  is  that 
the  organizational/managerial  distance  between  ETIP  and  the 
actual  experiment  was  so  large.  There  are  several  factors 
which  facilitated  or  hindered  this  structure. 

1.  The  commitment  of  the  Director  of  Regulatory  Standards 
at  AEC  was  a powerful  and  useful  factor  (even  though  project 
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monitoring  was  later  passed  to  someone  else) . Specifically: 

•The  Director  (Lester  Rodgers  at  the  time  of  project 
initiation)  was  a champion  of  reforming  the  standards  process, 
the  central  issue  of  the  experiment.  This  was  clear  from 
formal  contacts  between  ETIP  and  AEC  (Lewis,  Note  4;  Harter, 
Note  5)  and  his  testimony  to  Congress  (Rodgers,  1973). 

•The  Director  was  a member  of  the  Nuclear  Technical 
Advisory  Board  (NTAB)  of  ANSI,  the  group  responsible  for 
coordinating  nuclear  standards  work  at  ANSI.  He  was  also 
a member  of  the  executive  board  of  NTAB  and  later  took  part 
in  the  ad-hoc  committee  for  the  experiment  established  from 
ANSI/NTAB  membership. 

•The  Office  of  Regulatory  Standards  at  AEC  was  the 
immediate  user  of  ANSI  nuclear  standards  work  of  the  type 
to  be  used  in  the  experiment. 

2.  ETIP  is  interested  in  conducting  its  work  through 
other  government  agencies.  Contracting  the  experiment  through 
AEC  (rather  than  directly  with  ANSI)  assured  that  the 
Commission  had  a direct  stake  in  the  experiment.  However,  the 
control  of  project  funds  by  AEC  also  presented  some  problems 
for  ANSI  (see  section  on  implementation) . 
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3.  Essentially,  ETIP  did  not  have  control  over  the 
conduct  of  the  experiment,  as  this  rested  with  MTAB . This 
would  be  a problem  later  in  contracting  (see  next  section) 
and  experimental  methodology  (see  section  on  implementation) : 

•The  project  plan  left  a certain  flexibility  to  the 
program  manager  in  controlling  experimental  changes  in  the 
committees . 


•The  project  contracted  at  ANSI  was  already  different 
from  the  one  initiated  at  ETIP. 

4.  During  the  course  of  the  entire  experiment,  personnel 
in  the  chain  changed  (see  sections  on  implementation  and 
evaluation) . Also,  AEC  became  the  Nuclear  Regulatory  Commission 
(NRC)  in  1975. 

3.1.2  Contracting 

In  June,  1974,  the  funds  for  the  experiment  were  transferred 
to  AEC,  which  then  contracted  to  ANSI  on  a cost-reimbursable 
basis.  At  this  point  effective  control  from  ETIP  was  no 
longer  possible.  ANSI  went  out  for  bids  from  consulting  firms 
to  conduct  the  experiment,  contrary  to  the  plan  of  hiring  a 
program  manager  who  would  coordinate  ANSI  committees  and  in- 
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terventions.  ANSI  also  reworded  the  language  of  the  plan 
specifying  the  interventions  to  be  used.  These  interventions 
became  illustrative,  leaving  ANSI  the  option  to  suggest  other 
interventions  it  felt  important  (Harter,  Note  6) . These 
changes  were  not  expected  by  ETIP  (Harter,  Note  6) . 

The  result  of  these  arrangements  was  the  hiring  of 
Battelle-Northwest  to  conduct  the  experiment.  Later  a separate 
agreement  was  made  with  Harlan  J.  Anderson  from  Westinghouse- 
Hanford  Corporation  to  conduct  a similar  experiment  with  one  of 
his  committees,  using  only  one  of  the  proposed  interventions. 

The  proposal  by  Battelle  was  accepted  as  the  basis  for  their  work, 
3.2  Experimental  Design 

Some  amount  of  design  for  the  experiment  was  available 
before  the  experiment  began.  Battelle  discussed  independent 
variables,  the  variation  of  independent  variables  over  the  test 
committees,  and  the  dependent  variables.  On  the  other  hand, 
Anderson  only  proposed  the  one  change  variable  and  did  not 
discuss  others.  This  section  of  the  paper  will  focus  on  the 
.experimental  design  utilized  by  Battelle,  since  theirs  was 
more  complete  and  they  were  the  primary  contractor.  The  known 
characteristics  of  the  Anderson  experiment  will  be  identified 
as  the  Battelle  design  is  presented. 
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3.2.1  The  Approach  Used  and  Variables 

The  basic  approach  of  this  experiment  was  exploratory. 

It  was  hypotesized  that  certain  factors  in  the  drafting  process 
of  the  voluntary  standards  groups,  such  as  clerical  support 
and  prior  assignment  of  responsibilities,  would  have  a signi- 
ficant relationship  to  the  speed  at  which  a draft  of  a standard 
could  be  produced.  Other  considerations  were  the  cost  of 
alternative  processes  in  the  group  effort  and  +*he  quality 
of  output.  It  was  unclear  what  impact  new  committee  procedures 
would  have  on  these  two  variables. 

Independent  Variables 


Battelle  considered  factors  in  the  process  which  were  most 
likely  to  be  contributing  to  the  slow  development  of  standards 

(Battelle,  1975,  p.5).  The  independent  variables  chosen  for 

% 

manipulation  in  the  experiment  and  the  difference  over  normal 
procedure  were  as  follows  (Battelle,  1975,  pp.  6-7): 

1.  Working  meeting.  A five  day  working  meeting  held  at 
Battelle.  This  was  different  from  the  usual  practice  of  having 
several  periodic  meetings  and  communications  (ETIP , 1974,  p.ll). 

2.  Working  draft  or  outline.  A working  draft  of  some 
form  would  be  distributed  before  the  working  meeting.  While 
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this  appears  to  be  the  usual  procedure,  the  differences  here 
were  whether  it  was  a draft  or  outline. 

3.  Executive  secretary.  A support  person  responsible 
for  assuring  the  availability  of  needed  services  during 
meetings  and  revising  drafts  according  to  group  member  comments. 
This  constituted  new  services  for  meetings  and  a reduction  in 
the  chairman's  workload  in  communicating  committee  work.  This 
was  the  main  intervention  made  in  the  Anderson  experiment.  It 
appears  that  the  person  in  Anderson's  experiment  also  performed 
clerical  and  duplicating  functions,  a separate  intervention 

for  Battelle. 

4.  Clerical  and  duplicating  personnel.  Support  persons 
responsible  for  preparing  drafts  and  distributing  materials 
to  the  committee.*  This  reduced  some  of  the  work  normally 
done  by  the  chairman. 

5.  Payment  of  the  chairman.  Payment  of  the  chairman 
for  work  performed  on  the  standard.  This  was  different  from 
the  usual  practice  of  the  chairman  being  supported  through 
his  employer. 

6.  Travel  and  living  expenses.  Payment  to  committee 
members  for  travel  and  living  experses  incurred  while  par- 
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ticipating  in  meetings  on  standards.  This  was  different 
from  the  usual  practice  of  support  from  the  employers  of 
committee  members. 

7.  Preliminary  meeting.  Meeting  held  before  the  five- 
day  sessions  in  order  to  begin  preliminary  discussion  and 
assign  responsibilities.  This  intervention  appears  to  be 

due  more  to  recognition  of  group  meetings  before  the  experiment 
began . 

8.  Committee  membership.  Committee  sizes  were  limited 
to  about  ten  people.  While  this  also  appears-  to  be  a function 
of  group  conditions  before  the  experiment,  it  was  probably 

a reasonable  idea  to  keep  groups  the  same  size  so  that 
theorectically  the  workload  would  be  the  same  for  all  of  the 
committee  members. 

Battelle  used  these  factors  in  variation  on  three 
committees : 

1.  INMM  - 11  for  ANSI  standard  N15.28,  Criteria  and 
Standards  for  the  Certification  of  Nuclear  Materials  Managers. 

2.  ANS  - 2.12  for  ANSI  Standard  N635,  Guidelines  for 
Combining  Natural  Phenomena  and  Man-made  Hazards  at  Power 

I 
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Reactor  Sites,  and, 

3.  ANS  - 18.6  for  ANSI  Standard  N231,  Discharge  of  Thermal 


Effluents  Into  Surface  Water. 

Anderson  used  the  one  intervention  on  the  ASTM  C-26  Committee 
on  Fuel,  Control,  and  Moderator  Materials  for  Nuclear  Reactor 
Applications. 

The  conditions  were  varied  over  the  groups  as  follows 
(Battelle,  1975,  p.8): 


TABLE  I . Elements  of  the  Experiments 

Standard  Standard  Standard  ASTM 
N635  N15.28  N231  C-26 


Working  Meeting  X 

Outline  of  Standard  Prepared 
and  Distributed 


X . X 

X 


Draft  of  Standard  Prepared 
and  Distributed 

By  Chairman  X 


By  Executive  Secretary  X 

Executive  Secretary  Provided  X 

Clerical  and  Duplication 

Assistance  Provided  X 

Chairman  Paid 

.Travel  and  Living  Expenses  Paid 
Preliminary  Meeting/s  X 

X 


X 

X 

X 

X 

X 

X 


X X 

X X 

X 

X 


Committee  Size  Limited 


X 
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It  should  be  noted  that  independent  variables  were  not 
assigned  to  the  committees  in  any  apparent  procedure  other  than 
by  recognizing  already  existing  conditions  or  needs  (ETIP , 1974  , 
p.13) . It  was  to  be  the  duty  of  the  project  manager  to  assure 
that  they  were  "used  by  the  various  committees  in  such  a manner 
that  the  effect  of  each  can  be  appropriately  determined" 

(ETIP,  1974  , p . 13) . 

Also,  some  of  the  variables  as  listed  above  were  later 
ignored  in  the  data  analysis.  For  example,  the  specification  of 
draft  or  outline  was  reduced  to  working  draft,  and  a preliminary 
meeting  was  reduced  to  assignment  of  responsibilities  prior 
to  the  five-day  meeting. 

Dependent  Variables 


While  the  speed  of  the  drafting  process  was  the  major 
variable  of  interest  in  this  experiment,  there  were  two  other 
outcome  variables  which  also  needed  to  be  examined.  These  were 
the  quality  of  the  draft  standard  and  the  cost  to  produce  it. 
Because  of  the  many  groups  involved  in  the  standards  process 
(e.g.,  NTAB , ANSI,  NRC,  the  nuclear  power  community,  etc.), 
these  variables  were  further  defined  by  reference  to  the 
appropriate  group.  Thus  the  independent  variables  considered 
by  Battelle  in  their  proposal  were  (Battelle,  1974,  pp.  6-8): 
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1.  Time.  The  time  required  by  the  committee  to  produce 
a draft  for  submission  to  NTAB . 

2.  Quality  of  the  draft  standard.  The  quality  of  the 
draft  standard  based  on  the  number  arid  significance  of  the 
comments  received  from  the  field  on  the  first  draft  distributed 
for  comment  by  NTAB. 

3.  Cost  of  preparing  the  standard.  The  cost  of  prepara- 
tion (e.g.,  salaries,  travel,  clerical  support,  publication, 
etc.)  for  submission  to  NTAB. 

3.2.2  Analysis  of  the  Design 

In  order  to  evaluate  the  experimental  design  chosen 
by  Battelle,  some  criteria  were  needed*  One  scheme  which 
appeared  to  be  appropriate  was  the  use  of  a decision  process 
developed  by  Anderson  (1972) . In  this  process  experimental 
designs  are  chosen  on  the  basis  of  six  questions  about  what 
information  is  needed  from  the  experiment  and  the  conditions 
involved.  This  diagram  and  supporting  papers  appear  in  the 
appendix. 

*Most  of  the  following  comments  also  apply  to  the  experiment 
conducted  by  Anderson  for  ASTM  C-26. 
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The  six  questions  are  listed  below  and  the  answers,  as 
determined  by  the  writer,  follow: 

1.  Which  type  of  evaluation  is  needed? 

Although  the  approach  to  this  experience  was  explora- 
tory, the  independent  and  dependent  variables  were  known  and 
some  of  the  expected  outcomes  were  discussed  before  the 
experiment.  The  evaluation  needed  to  assess  the  effects  of 
the  independent  variables  on  the  outcome  variables:  time,  cost 

and  quality. 

2.  Can  control  groups  be  used? 

Control  groups  were  used  for  the  experiment  in  that 
three  standards  each  requiring  a different  committee  were  chosen 
and  the  independent  variables  varied  in  the  treatments . However , 
the  indpendent  variables  were  numerous  and  not  assigned  in  any 
logical  order  for  control  purposes.  As  it  turned  out,  each 
of  the  three  Battelle  groups  had  almost  all  of  the  possible 
changes.  This  problem  was  identified  to  Battelle  early  in  the 
experiment  by  NTAB  members,  apparently  with  no  effect. 

(Nuclear  Technical  Advisory  Board,  1974,  p.7)  . 

Another  weakness  with  the  design  was  that  there  were  no  control 
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groups  operating  under  the  usual  circumstances.  This  need  was 
considered  in  the  ETIP  project  plan  (ETIP,  1974,  p.15).  How- 
ever, further  contact  with  the  committee  system  at  ANSI  indicat- 
ed the  difficulty  of  comparing  different  people,  technology 
issues,  and  priorities  (in  terms  of  need  by  NRC)  (Harter,  Note  1) . 

3.  Is  random  assignment  possible? 

Random  assignment  of  groups  to  the  experimental  con- 
ditions or  to  the  usual  arrangements  did  not  occur.  It  was 
probably  not  feasible  given  that  NTAB  was  to  ‘select  standards 
needing  priority  attention,  having  established  drafting  committee 
chairman  but  no  previous  meetings,  covering  different  fields, 
and  having  similar  levels  of  difficulty  (ETIP,  1974,  p.12. 

Nuclear  Technical  Advisory  Board,  1974,  p.6). 

* 

Random  assignment  of  committees  to  the  different 
treatment  conditions  was  not  performed  either  because  it  was 
felt  by  ETIP  and  NASI  that  the  committees  should  have  the 
flexibility  to  choose  conditions  based  on  need  (ETIP,  1974, 
p.13 , Note  6) . 

4.  Are  repeated  measurements  possible? 


Repeated  measurements  would  have  provided  some  before 
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and  after  treatment  comparisons  of  committee  performance. 
However,  the  committees  were  typically  formed  to  draft  one 
specific  standard.  This  prevented  any  comparisons  between 
committee  performance  on  two  different  standards,  especially 
with  main  interest  being  on  variables  based  on  committee 
output  only  (time,  cost,  and  quality) . 

Given  the  answers  to  these  questions,  and  the  flow  of 
the  decision  process  of  Anderson's  (1972,  p.50),  the  result- 
ing conclusion  is  that  design  3 from  Campbell  and  Stanley 
(1966)  would  have  been  the  design  appropriate  to  the  conditions 
of  the  experiment.  This  design  uses  one  set  of  observations 
made  on  two  comparable  but  not  randomized  groups : 


X 0^  X - treatment  (application  of  the 

independent  variables) 

O2  0 = observation  of  the  dependent 

variables 

> 

TIME 

The  design  used  by  Battelle  was  weak  on  two  points  then: 

1.  They  did  not  utilize  any  comparison  groups  operating 
under  the  normal  working  committee  conditions.  This  would 
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make  it  difficult  to  validity  claim  improvements  over  normal 
conditions . 

i 

‘ 

2 . They  did  not  vary  the  treatment  or  reduce  the  number 
of  treatment  options  to  the  point  where  the  three  experiment 
groups  could  act  as  controls  to  each  other.  Thus,  the  inde- 
pendent effects  of  the  different  treatment  options  would  be 
unidentifiable . 

Some  other  problems  arising  due  to  these  conditions  were: 

1.  Selection. 

Since  randomized  assignment  of  groups  was  not  possible 

and  treatments  were  used  according  to  need,  it  would  be 

' 

difficult  to  identify  varying  committee  experiences  as  being 
the  result  of  the  varying  treatment  conditions  or  the  self- 
selection of  these  conditions  by  the  committees. 

i 

I I 

2.  Interaction  of  selection  and  the  treatment. 

Without  comparisons  to  groups  operating  under  the  usual  arrange- 

1 I 

ments , it  would  be  difficult  to  claim  any  generalizability  to 
the  results.  Experimental  results  could  be  partly  or  solely 
a function  of  the  types  of  committees  chosen  for  the  experi- 


ment . 
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3.3  Implementation  of  the  Experiment 

Battelle  started  work  on  the  contract  in  October,  1974,  and 
submitted  their  report  one  year  later  in  September  1975.  A sep- 
arate agreement  was  made  with  Harlan  J.  Anderson  of  Westinghouse- 
Hanford  Corporation  in  November  1974,.  and  a final  report  received 
in  February  1976.  Since  all  control  of  the  experiment  was  essen- 
tially at  ANSI,  there  is  little  to  discuss  about  the  implementation 
from  the  ETIP  perspective*.  The  implementation  by  ANSI  is  covered 
in  other  sections  on  planning  and  analysis. 

There  were,  however,  several  interesting  events  that  occurred 
during  the  contract  period  which  should  be  noted.  These  serve  to 
illustrate  the  problems  with  the  contractual/organizational  setup 
of  the  experiment: 

•An  ad  hoc  committee  was  established  from  ANSI -NT AB  members 
to  oversee  the  experiment.  No  ETIP  representative  was  asked  to 
participate  until  January  1975,  at  which  time  one  person  acting  as 
an  ex  officio  member  was  requested.  This  delay  meant  that  ETIP 
had  no  direct  input  to  the  proposal-contract  award  decisions. 

•Lester  Rodgers,  the  Director  of  the  Regulatory  Standards 
Office  at  AEC,  retired  at  the  end  of  1974.  Rodgers  was  a 

* 

This  lack  of  control  by  ETIP  later  increased  ETIP ' s interest 
in  conducting  a more  well  designed  evaluation  and  re-estab- 
lishing some  control  . (Harter , Note  1). 
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champion  of  standards  refroms  and  helped  immensely  in  getting 
the  experiment  initiated.  However,  he  did  not  participate 
in  the  day-to-day  monitoring  during  the  experiment.  There 
are  mixed  feelings  at  ETIP  whether  this  had  an  impact  or 
not.  For  example,  the  main  contact  at  the  Regulatory  Standards 
Office  remained  the  same,  and  the  new  director  had  similar 
reform  ideas  to  Rodgers'  (Minogue,  1974a,  1974b).  On  the 
other  hand,  negotiations  with  NRC  were  not  as  smooth  for  the 
evaluation  contract  later,  and  NRC  lost  interest  in  having 
contracting  responsibility  for  the  evaluation.  (See  section 
on  evaluation) . 

•Also,  in  January  1975,  an  executive  order  from  the 
White  House  (based  on  the  Energy  Reorganization  Act  of  1974) 
was  issued  establishing  the  Nuclear  Regulatory  Commission 
(NRC)  from  parts  of  the  Atomic  Energy  Commission  (AEC) . It 
is  felt  at  ETIP  that  this  changeover  had  litte  impact  on  the 
experiment,  since  the  same  people  remained  on  the  project 
and  the  new  Office  of  Standards  Development  was  essentially 
the  same  as  the  previous  Office  of  Regulatory  Standards. 

•By  January  1975,  three  months  after  the  beginning 
of  the  Battelle  contract,  problems  with  mechanics  of  the 
contract-agreement  with  ETIP,  NRC,  and  ANSI  were  starting  to 
arise.  NRC  by  charter  was  not  allowed  to  use  any  money  except 
that  appropriated  by  Congress  (Penn,  Note  7) . This  necessitated 


29 


an  arrangement  where  ANSI  had  to  bill  NRC  who,  in  turn,  billed 
the  NBS  account.  For  ANSI,  this  procedure  was  different  from 
their  usual  one  of  operating  with  grants  (Van  Wyk,  Note  8) . 

ANSI  needed  faster  processing  of  bills  and  also  wanted  reim- 
bursement of  precontract  costs  (possibly  arising  from  misunder- 
standing the  meaning  of  a cost-reimburseable  contract) . NRC 
held  bills  until  a large  enough  amount  had  accumulated. 

3.4  Analysis  of  Outcomes 

•Two  final  reports  were  issued  for  this  experiment,  one 
by  Battelle  (Wittenbrock , 1975)  for  three  test  committees, 
and  one  by  Anderson  (1976)  for  one  test  committee.  These 
reports  are  completely  different  and  will  be  discussed  in 
separate  sections  here. 

3.4.1  The  Battelle  Report 

Battelle  proposed  to  analyze  the  experiment  on  the  basis 
of  time  for  preparation,  quality  of  standards  as  determined 
by  comments  from  relevant  groups,  and  cost  (Battelle,  1974,  p.2) 
At  the  same  time  they  indicated  that  the  limitations  of  the 
project,  and  the  large  number  of  variables  involved  would 
provide  little  more  than  qualitative  results  (Battelle,  1974, 
p.3) . They  proposed  to  issue  a questionnaire  to  the  test 
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committee  members  for  their  evalution  of  the  experimental 
approaches.  Also  a "check  sheet"  would  be  used  to  summarize 
the  effectiveness  of  the  three  different  treatment  approaches 
(Battelle,  1974,  p.8). 

X 

However,  Battelle  did  not  present  in  their  final  report 
most  of  what  they  discussed  in  their  proposal.  Again,  argu- 
ments about  the  lack  of  quantitative  measures,  limited  time, 
limited  data,  and  many  variables  were  discussed  to  justify 
a different  approach  to  the  analysis  of  the  experiment 
(Wittenbrock,  1975,  p.13).  The  final  evaluation  consisted 
of  ratings  from  group  members  on  the  effectiveness  of  the 
innovations,  observations  by  the  Battelle  project  manager, 
and  costs  for  the  five-day  working  meetings.  Thus,  it  should 
be  noted  that  the  three  dependent  variables  were  either  not 
examined  or  only  superficially  so.  Specifically: 

1.  Time  to  prepare  the  draft. 

Time  for  preparation  originally  was  defined  in  terms 
of  the  period  up  until  submission  of  the  draft  to  NTAB . ' Al- 
though, theoretically , this  time  length  was  available  to 
Battelle  as  they  were  involved  up  to  these  draft  submissions, 
no  discussion  of  time  was  presented. 


2.  Quality  of  the  draft. 
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Originally,  draft  quality  was  defined  in  terms  of  the 
number  and  significance  of  the  comments  received  from  the  field 
on  the  draft  submitted  to  NTAB  (Battelle,  1974,  p.6).  Quality 
in  these  terms  was  not  discussed  in  the  final  report. 

3.  Cost  of  preparing  the  draft. 

Originally,  cost  estimates  were  to  include  all  expenses 
of  the  standard  drafting  process.  The  costs  discussed  in  the 
final  report  only  include  the  costs  of  the  executive  secretary 
and  salaries  of  the  chairman  (Wittenbrock , 1975,  p.18).  Travel 
and  living  expenses  were  not  presented.  The  expenses  of  other 
committee  meetings  besides  the  ones  at  Battelle  were  also  not 
included.  All  of  the  committees  held  meetings  other  than  the 
five  day  session  at  Battelle. 

Finally,  the  evaluation  "check  sheet"  which  was  to  act  as 
a summary  of  the  effectiveness  of  the  three  different  approaches 
was  not  included. 

Ratings  by  Committee  Members 


Questionnaires  were  distributed  to  committee  members  at 
or  near  the  completion  of  the  working  session  at  Battelle.  It 
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was  felt  by  Battelle  that  because  committee  members  had  no 
long  term  commitment  to  the  committee,  their  feedback  would  be 
relatively  unbiased.  While  this  shouldn't  be  disputed,  the 
weakness  was  that  their  comments  received  the  most  attention 
in  the  report. 

The  questionnaires  had  16  items  which  were  identical  for  all 
test  committees,  except  for  tense  in  order  to  account  for 
treatment  variations.  All  members  were  asked  to  rate  items  as 
experienced  or  as  they  would  have  experienced  them.  This  was  a 
fairly  weak  approach  given  the  weight  placed  on  the  results  and 
the  analysis  techniques  used. 

The  16  items  obtained  ratings  via  Likert  scales  for  the 
following  issues  (Wittenbrock , 1975,  p.14): 

1.  The  usefulness  of  each  of  the  specific  innovations 
instituted  in  this  project; 

2.  The  extent  to  which  the  composition  of  the  committee 
was  appropriate  for  the  task; 

3.  The  skills  of  the  chairman; 

4.  The  difficulty  of  the  standard  in  question. 
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(e .g. , scope) ; and 

5.  The  overall  performance  of  the  committee. 

rt  should  be  noted  that  only  seven  innovations  were  con- 
sidered in  these  questionnaires,  where  eight  had  originally 
been  defined  (Wittenbrock , 1975,  pp.6-7).  Committee  membership, 
while  presented  as  an  innovation,  was  dropped.  Also,  there  was 
some  modification  of  innovation  definitions.  For  example, 
preliminary  meeting  was  changed  to  mean  prior  assignment  of 
responsibilities,  leaving  out  prior  organization  and  discussion 
as  originally  specified. 

The  first  step  of  item  analysis  was  to  present  the  average 
rating  for  each  innovation.  The  means  for  each  group  and  the 
groups  combined  were  high  (favorable) , except  for  prior  assign- 
ment of  responsibility,  which  had  a more  unfavorable  rating. 
Battelle  neglected  to  present  standard  deviations  and  ranges 
on  these  ratings,  common  and  necessary  indicators.  There  was 
no  way  to  gauge  the  level  of  agreement  within  the  groups.  Also, 
they  presented  a mean  of  all  of  the  individual  innovation  means, 
a meaningless  indicator. 

Next,  these  means  were  subjected  to  an  analysis  of  variance 
to  indicate  agreement  across  groups.  Three  of  the  innovations 
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had  group  mean  ratings  which  were  statistically  different 
significantly  from  each  other.  While  the  mean  ratings  would 
seem  to  support  these  results,  it  would  have  been  useful  to 
see  significance  levels.  Battelle  did  not  indicate  what  it 
considered  significant.  Also,  Battel'le  made  no  effort  to  discuss 
why  these  differences  existed. 

The  report  then  discussed  another  eight  variables, 
basically  parameters,  which  reflected  the  working  characteris- 
tics of  the  committees , such  as  adequacy  of  committee  represen- 
tation and  availability  of  data  needed  for  writing  the  standard. 
Again,  as  with  the  innovation  ratings,  means  are  presented  for 
for  each  group  and  overall.  There  are  no  standard  deviations 
and  ranges.  The  analysis  of  variance  using  these  means  did 
not  provide  any  significant  differences,  but  the  actual  results 
are  not  presented.  Overall  performance,  the  last  variable, 
is  handled  in  the  same  way  as  these  also. 

The  next  analysis  discussed  were  regressions  of  the  innova- 
tive and  committee  characteristic  variables  on  overall  perfor- 
mance. First,  the  innovative  ratings  were  regressed  on  overall 
performance , then  the  committee  characteristic  ratings , and  then 
both  of  these  sets  were  regressed  on  overall  performance.  The 
prior  assignment  of  responsibility  had  the  only  statistically 
significant  relationship  with  overall  performance.  The  inter- 
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pretation  was  that  prior  assignment  was  a "good"  predictor  of 
the  group  performance  rating — the  more  favorable  the  rating 
towards  prior  assignment,  the  more  favorable  the  rating  of 
overall  performance.  It  would  have  been  interesting  to  examine 
this  result  in  light  of  the  previous  indications  that  prior 
assignment  was,  on  the  average,  the  lowest  rated  innovation. 

There  are  several  problems  common  to  all  of  these  regres- 
sions. These  are: 

1.  Small  sample  size,  too  many  variables. 

The  three  groups  had  sizes  of  6,  7,  and  10,  or  23  total. 
While  these  are  in  general  small  sizes  on  which  to  base  conclu- 
sions, this  weakness  is  even  more  critical  in  the  regression 
models  used,  where  the  number  of  variables  in  the  analysis 
approaches  the  number  of  individual  cases  (Weiss,  Note  9;  Draper 
and  Smith,  1966,  pp.  62-63).  For  example,  a perfect  fit  for  a 
regression  line  can  be  made  when  there  are  two  data  points  and 
one  independent  variable.  Thus,  n-1  variables,  where  n is  the 
total  number  of  cases,  can  specify  a perfect  fit.  In  the  * 
Battelle  regressions,  the  three  analyses  are  7 variables  and  23 
cases,  8 variables  and  23  cases,  and  15  variables  and  23  cases. 

2 

The  R,  factor  in  regression,  a measure  of  the  propor- 
tion of  variation  in  data  explained  by  a regression  line,  is 
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affected  by  this  small  sample  problem.  Since  the  impact  is  to 
inflate  R2 , one  is  even  less  confident  that  Battelle  really 
found  anything  significant. 

2.  Lack  of  specific  regression  information. 

Many  of  the  indicators  normally  presented  with  regression 
analyses  are  not  presented  in  the  report.  Most  importantly, 
analysis  is  presented  for  the  innovation  variables  only.  This 
included  statistical  significance  levels  for  each  innovation 
and  the  respective  increases  in  R2  as  they  were  entered  in  the 
model.  The  report  does  not  include  any  discussion  of  the 
regression  constants  A (intercept)  or  B (slope) , which  indicates 
the  relative  importance  of  each  independent  variable.  These 
are  normally  tested  to  see  if  they  are  significantly  different 
from  zero.  Also,  *the  standard  errors  of  the  B constants  were 
not  included,  providing  no  indication  of  the  variability  of 
the  estimate. 

3.  No  overall  regression  test. 

A standard  test  of  any  regression  line  is  the  overall 
F test  of  whether  R (from  R2)  equals  zero,  or  equivalently, 
whether  all  B weights  are  equal  to  zero.  This  test  is  not 
discussed.  The  "F  to  enter"  statistic  presented  is  not  ex- 
plained. 
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The  combined  effect  of  all  of  these  problems  in  presentation 
and  analysis  is  a decrease  in  confidence  with  the  results.  The 
two  major  findings  based  on  these  analyses  are: 

1.  That  most  innovations  tended  to  have  positive  effects, 
but  several  of  them,  such  as  payment  of  chairmen  and  living 
expenses,  did  not,  and 

2.  That  prior  assignment  of  responsibilities  seemed  to  be 
related  to  overall  performance. 

These  have  little  support.  A more  rigorous  approach  in 
the  application  of  changes  and  comparison  to  other  committees 
is  needed. 

Observations 

Observations  of  the  three  committees  were  made  by  the 
project  manager  at  Battelle  and  apply  mostly  to  the  five  day 
workshop  period  at  Battelle.  They  support  the  favorable  ratings 
given  by  committee  members , but  do  not  add  a lot  of  weight 
to  the  results . 

Summary 


The  basic  conclusions  to  be  drawn  from  the  Battelle  analysis 
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are  that: 

1.  The  analysis  did  not  fulfill  the  proposed  effort,  and 

2.  While  the  experiment  met  some  of  the  needs  for  explora- 
tory information,  no  resolution  was  possible  on  the  effects 

of  the  individual  innovations. 

' • 1 

3.4.2  The  Anderson  Report 

The  analysis  of  the  ASTM  C-26  committee  performance  was 
done  independently  of  the  Battelle  report  and  submitted  several 
months  later  (Anderson,  1976).  Only  one  of  the  innovations 
was  utilized,  that  of  providing  a knowledgeable  executive 
secretary.  This  basic  result  claimed  was  that  the  provision 
of  a secretary  sped  up  the  committee  activities. 

For  the  dependent  variables  used  by  Battelle  results  were: 

1 . Time . 

Two  standards  were  produced  within  one  year  instead 
of  the  usual  two  to  three.  (Anderson,  1976,  p.3).  Some  examples 
were  provided  to  show  this,  but  these  were  not  organized  well 
enough  in  proving  the  point. 
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2.  Cost. 

The  cost  of  the  executive  secretary  was  presented  in 
detail.  However,  the  salary  for  the  secretary  was  not  included 
since  it  was  reasoned  that  under  the  ideal  system,  this  salary 
would  be  "donated"  by  the  companies  or  organizations  supporting 
the  committee  members  (Anderson,  1976,  p.l) . 

3.  Quality. 

There  was  no  definitive  discussion  of  quality  other 
than  the  aforementioned  reference  to  appendix  reports.  To  the 
unfamiliar  observer,  the  evidence  provided  was  unclear  and  not 
persuasive.  On  the  other  hand,  the  C-26  committee  appeared  to  be 
enthusiastic  about  the  results  of  using  the  executive  secretary, 
implying  satisfaction  with  the  product  (Anderson,  1976, 
pp.  3-4) . ♦ 

While  the  experiment  intended  with  this  committee  was 
simpler  than  that  of  Battelle,  it  did  not  offer  a level  of 
comparability  to  Battelle,  whereby  commonalities  could  be 
assessed.  The  parameters  briefly  considered  by  Battelle^  do 
not  appear  at  all  in  discussions  by  Anderson.  The  overall  per- 
formance of  C-26  is  not  discernable  from  the  report. 
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3.5  Evaluation  of  the  Experiment 

An  evaluation  of  the  NRC  experiment  was  first  considered 
in  the  plan  initiating  the  project  (ETIP,  1974,  pp.  14-16). 

It  was  proposed  that  ETIP  hire  a separate  contractor  for  this 
purpose.  The  content  of  an  evaluation  was  only  briefly  con- 
sidered. Further  delineation  of  the  evaluation  needs  at  this 
stage  might  have  evolved  into  an  "evaluability  assessment", 
possibly  guiding  the  design  enough  to  avoid  some  of  the 
problems  encountered  later. 

3.5.1  Evaluation  Project  Plan 

In  the  original  project  plan  several  evaluation  issues 
were  discussed.  These  included  the  timeliness  of  the  process 
attempted,  the  quality  of  the  drafts  in  the  opinion  of  AEC, 
the  efficient  and  effective  use  of  committee  members,  and  the 
comparison  of  acceleration  accomplished  versus  costs.  Con- 
sidering the  lack  of  background  (at  the  time  of  the  plan)  on 
the  standards  process  from  ANSI  through  NRC,  it  is  unclear 
whether  these  questions  could  have  been  answered  in  a reason- 
able amount  of  time.  For  example,  the  experiment  essentially 
ended  at  the  draft  review  process  in  ANSI,  but  the  evaluation 
was  to  look  at  impacts  in  NRC. 

During  the  experiment,  it  became  increasingly  clear  that 
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the  evaluation  would  have  to  be  more  encompassing  that  the 
above  considerations.  This  arose  because  ETIP  and  ANSI  per- 
sonnel felt  that  the  design  by  Battelle  was  leaving  many 
factors  unconsidered  (Harter,  Note  1).  Also,  ETIP  had  re- 
quested the  Technical  Analysis  Division  (TAD)  of  NBS  to  review 
the  experiment  and  propose  methods  for  measuring  variables 
(Harter,  Note  1).  TAD  commented  that  the  intervention  im- 
pacts were  already  obvious,  that  the  design  was  poor  and  left 
variables  such  as  group  dynamics  or  technical  problems  uncon- 
sidered, and  that  there  were  too  many  variables  for  effective 
control  (Swisher,  Note  3;  Harter,  Note  10). 

The  outcome  of  these  events  was  a more  encompassing  project 
plan  for  evaluation  that  was  adopted  by  the  Department  of 
Commerce  in  April,  1975  (midway  through  the  experiment).  The 
plan  considered  backgrounding  of  the  standards  development 
process,  a detailed  review  of  the  experiment,  the  post-experi- 
ment  history  of  the  drafts,  the  project  costs,  and  what 
recommendations  could  be  made  (ETIP,  1975).  The  plan  still 
lacked  any  emphasis  on  comparing  the  process  and  outcomes  of 
the  experiment  with  other  similar  variables  in  the  normal 
procedure  used  at  ANSI.  Another  problem  with  the  plan  yas 
that  it  concentrated  heavily  on  process  variables.  With  the 
experiment  half  over  already,  it  could  not  be  expected  that 
the  perishable  process  data  would  be  obtainable. 
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3.5.2  > Ontracting  the  Evaluation 

With  the  evaluation  depending  on  capturing  the  process 
experience  of  the  committees,  the  necessity  of  contracting  the 
work  as  soon  as  possible  was  obvious.  However,  the  plan  was 
seriously  delayed  and  timely  access  to  the  participants  and 
results  of  the  experiment  lost. 

The  basic  organizational  arrangement  for  the  evaluation 
was  to  have  been  the  same  as  with  the  experiment.  Funds  were 
transferred  to  NRC  in  June,  1975,  but  it  wasn’t  until 
December  1975,  three  months  after  Battelle  issued  its  final 
report,  that  NRC  sent  a letter  to  ANSI  to  prepare  a proposal 
for  evaluation.  This  work  was  designated  by  NRC  as  an  exten- 
sion of  the  experiment  contract. 

ANSI  was  to  have  responded  by  the  end  of  1975,  but  they 
took  until  February,  1976.  During  this  time  ANSI  sent  an  RFP 
out  for  the  evaluation  contract  and  received  three  responses 
(Stafco,  1976,  Rockwell  International,  1976,  Nuclear  Services 
Corp.,  1976).  Also,  the  Director  of  Nuclear  Programs  at 
ANSI , who  had  been  the  main  contact  and  coordinator  for  the 
experiment,  left  ANSI. 

Before  NRC  received  the  ANSI  response  in  February,  ETIP 
contacted  NRC  and  expressed  displeasure  with  the  process 
occurring  (Berlin,  Note  11) . A month  later,  in  March,  there 
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was  still  no  contract  with  ANSI.  Also,  several  new  events 
began  to  reorient  the  project. 


•The  new  coordinator  at  ANSI  for  the  experiment  indicated 
that  the  process  studied  in  the  project  was  a small  part  of 
the  entire  standards  process.  He  wished  to  explore  the  other 
parts  and  develop  new  ideas  during  the  evaluation  project. 

•ETIP  discussed  the  evaluation  mostly  with  ANSI  personnel, 
leaving  NRC  unaware  of  the  changes  being  considered. 

•ETIP  began  to  push  for  comparing  the  tested  committees 
to  others  operating  under  normal  procedures.  ETIP  also 
pushed  for  a system  analysis  of  the  entire  standards  process 
to  gain  some  perspective  on  the  experiment. 


•NRC  attempted  to  put  through  a sole-source  justification 
for  contracting  with  ANSI.  This  was  running  into  delays  be- 
cause NRC  was  uncertain  that  ANSI  was  uniquely  capable  to  con- 
duct the  evaluation . * 

*This  was  a reasonable  concern  given  that  ANSI  was  not  uniquely 
qualified  to  do  evaluation.  Also,  with  more  and  more  emphasis 
being  placed  on  follow-up  work  in  the  standards  process,  ANSI 
would  potentially  be  investigating  the  process  at  NRC.  This 
would  not  have  been  an  appropriate  arrangement,  especially 
considering  NRC ' s reluctance  to  have  anyone  study  their 
processes . 
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The  outcome  of  these  events  was  that  ANSI  issued  a new 
proposal  to  NRC  in  June,  1976,  but  NRC  could  still  not  contract 
the  evaluation  to  them. 

At  a meeting  in  July,  1976,  NRC  expressed  the  desire  to 
remove  themselves  from  contracting  responsibility.  While 
they  were  still  interested  in  evaluation,  they  felt  that 
ETIP  could  more  appropriately  manage  it  (Garrity,  Note  12) . 

The  situation  was  also  further  complicated  by  the  fact  that 
the  original  contract  between  NRC  and  ANSI  expired. 

ETIP  eventually  agreed  to  a return  of  the  funds  for 
evaluation  from  NRC.  The  evaluation  project  was  deobligated 
in  November  1976,  and  the  original  experiment  was  deobligated 
in  December  1976. 
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4.0  Conclusions  and  Recommendations 

4 . 1 The  NRC  Experiment 

The  results  of  the  experiment  are  fairly  inconclusive. 
While  the  work  of  the  selected  committees  may  have  been 
accelerated  by  the  various  interventions,  it  is  unclear  how 
the  improvements  compare  to  normal  circumstances,  and  what 
the  overall  impacts  have  been  on  the  drafting  process  at 
ANSI.  The  analyses  presented  are  insufficient  and  at  times 
methodologically  weak  for  determining  the  individual  impacts 
of  the  different  changes  made.  If  any  conclusion  can  be  made 
about  the  results,  it  is  that  it  appears  reasonable  to  accel- 
erate the  committee  work  by  providing  extensive  support  ser- 
vices in  combination  with  prolonged  meetings.  What  impor- 
tance this  has  toother  committees,  other  problems  in  drafting 
standards,  other  parts  of  the  process,  or  industrial  technology, 
is  unknown. 

The  option  still  exists  at  this  time  to  contract  some 
analysis  of  the  experiment  in  the  evaluation  phase  of  the 
project.  Clearly  the  process  data  so  important  to  this  experi- 
ment is  no  longer  obtainable.  However,  the  resulting  draft 
standards  probably  exist  somewhere  in  the  standards  process 
and  these  may  be  interesting  to  examine.  For  example,  the 
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history  of  each  draft  as  it  went  through  further  review  may 
be  able  to  provide  some  evidence  of  the  quality  of  the  draft, 
one  of  the  original  outcome  variables.  The  history  may  also 
show  the  relative  importance  of  the  acceleration  of  committee 
work  to  the  later  review  processes  of  ANSI  and  NRC. 

If  any  work  is  to  be  done,  some  effort  should  be  included 
on  specifying  what  constitutes  the  standards  process.  This 
would  at  least  add  some  perspective  to  the  experiment  and  could 
also  point  out  other  areas  where  further  work  could  be  done. 

4.2  ETIP  Experiments 

It  seems  clear  from  this  experiment  that  the  backgrounding 
and  contracting  phases  of  a project  are  two  of  the  more  impor- 
tant considerations.  While  the  hindsight  review  of  these 
factors  easily  demonstrates  their  weakness,  it  does  need  to  be 
realized  that  items  overlooked  at  these  initial  stages  can  have 
impacts  on  later  parts  of  the  experiment  that  can  not  be  so 
easily  controlled  or  adjusted.  For  example,  the  apparent 
lack  of  perspective  of  the  ANSI  functions  in  the  overall  standards 
process  left  some  uncertainty  as  to  the  importance  of  the 
experiment.  Several  groups  expressed  this  point  during  the 
project  and,  in  fact,  the  lack  of  importance  became  one  of  the 
lasting  conclusions  about  this  work. 


48 


What  is-  needed  in  backgrounding  is  the  careful  and  compre- 
hensive examination  of  groups  having  a stake  (either  directly 
or  indirectly)  in  the  experiment.  This  includes  defining  the 
problem  (or  lack  of)  as  each  group  views  it  and  identifying 
what  they  might  get  out  of  the  experiment.  This  is  especially 
important  where  cooperation  of  a group  is  needed.  Specifica- 
tion of  ETIP  contributions  and  utility  in  the  project  should 
also  receive  full  attention. 

Of  equal  importance  with  these  analyses  is  the  arrange- 
ments made  between  the  groups  to  accomplish  their  objectives. 
The  most  obvious  problem  of  the  NRC  experience  was  the  organi- 
zational distance  between  ETIP  and  the  experiment.  The  lack 
of  control  for  ETIP  in  this  arrangement  resulted  in  some 
serious  methodological  problems  in  the  experiment  of  large 
importance  to  ETIP.  The  arrangement  was  also  not  suitable  for 
the  two  phases  of  the  project.  While  ANSI  control  was  most 
likely  appropriate  for  management  of  the  interventions,  it  was 
less  relevant  to  an  evaluation  of  the  changes.  This  was 
especially  true  from  the  viewpoint  of  NRC  since  the  ANSI  work 
would  have  involved  examining  their  processes. 

The  use  of  two  phases , one  for  the  interventions  and  one 
for  the  evaluation,  also  needs  some  discussion.  First,  it  is 
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useful  to  have  a follow-on  strategy  for  evaluation,  as  this 
project  did  to  some  extent.  However,  this  should  be  carried 
even  further  to  the  point  of  assessing  the  evaluability  of  any 
changes  before  they  are  made.  An  "evaluability  assessment" 
would  serve  to  further  tie  the  interventions  and  evaluations 
to  each  other.  This  need  was  clearly  demonstrated  in  this 
case  because  of  the  major  emphasis  on  process  variables. 
Evaluation  at  the  later  point  could  not  compensate  for  the 
original  design  deficiencies. 

Furthermore,  it  would  seem  useful  that  ETIP  consider  an 
experiment  as  consisting  of  interventions  and  evaluation 
rather  than  separating  the  two  into  several  independent 
phases.  Even  a cursory  glance  at  the  Battelle  work  demon- 
strates the  need  for  this,  even  if  one  only  considers  that 
the  difference  between  Battelle' s "experimental  analyses"  and 
the  proposed  second  phase  evaluation  was  small.  In  principle 
there  is  no  difference  and  the  NRC  experience  demonstrates 
the  problems  of  trying  to  separate  them. 
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A minus  indicates  a weakness;  a plus  indicates  that  the  design  controls  for  that  threat;  a question 
mark  indicates  a possible  source  of  concern;  and  a blank  indicates  that  the  factor  is  irrelevant. 

(from  Campbell  and  Stanley,  1966,  pp.  8,  40,  and  56) 
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Table  3.1 

Strengths  and  Weaknesses  of  Research  Designs 
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Campbell  and  Stanley  (1966)  discuss  16  social  science  research 
designs  and  describe  which  of  the  above  threats  to  validity  each  falls 


prey  to.  Their  list  of  research  designs  is  as  follows: 

1.  One  shot  case  study:  a single  group  is  studied  only  once, 
after  exposure  to  the  treatment  of  interest  — 

X 0 

treatment  observation 

2.  One  group,  pretest  and  posttest:  a single  group  is  observed 

before  and  after  — 

0X0 


3.  Static  group  comparison:  two  groups  are  studied  (one  after 

treatment,  one  after  no  treatment)  — 


X ox 
02 

4.  Pretest-Posttest  Control  group  design:  two  randomly  constructed 

groups  are  studied  (one  before  and  after  treatment,  one  before 
and  after  no  treatment)  — 

R 0X  X 02 

°3  °4 

randomized. 


5.  The  Solomon  Four-group  design:  four  randomly  constructed  groups 

are  studied  (two  before  and  after,  one  not  receiving  treatment, 
and  two  after,  one  not  receiving  treatment)  — 


R 0X  X 02 
R O3  O4 

R X 05 

R 06 


6. 


The  Posttest-only  control  group  design:  two  randomly  constructed 

groups  are  studied  (one  after  treatment,  one  after  no  treatment)  — 


R X 0X 

R 02 

7.  The  time-series  experiment:  one  group  is  measured  repeatedly, 

a treatment  being  administered  sometime 


0 


n 


°1  °2 


X 
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The  equivalent  time-samples  design:  one  group  is  measured  with 

randomly  alternating  exposure  to  similar  occasions  — 


XqO 


X ]_0  Xo0  Xx0 

7 

Occasion  with  treatment  present 

Occasion  without  treatment  present 


The  equivalent  materials  design:  one  group  is  measured  repeatedly 

with  exposure  to  equivalent  samples  (a-d)  of  materials  — 


MaXl0  M^O  McX_!0  MdXo0 
materiality 
sample 

10.  The  nonequivalent  control  group  design:  two  groups  (not  pre-experimentally 

equivalent)  are  measured  before  and  after  — 

0X0 
0 0 

11.  Counterbalanced  design:  two  or  more  groups  are  exposed  to  different 
treatments  at  different  times  — 


Time  1 

Time  2 

Time  3 

Time 

Group  A 

xxo 

x2o 

X30 

x4o 

Group  B 

X20 

X4O 

xxo 

X3O 

Group  C 

x3o 

X-lO 

x40 

X20 

Group  D 

x40 

X30 

X20 

xxo 

12.  The  separate- sample  pretest-posttest  research  design:  two  randomly 
chosen  groups  are  measured  (one  initially,  one  after  treatment)  — 

R 0 (X)  irrelevant  treatment 

R X 0 

13.  The  separate-sample  pretest-posttest  control  group  design:  four 

randomly  chosen  groups  are  measured  (two  initially,  one  after  treatment, 
one  after  no  treatment)  — 


R 0 
R 
R 
R 


0 


0 


(X) 

X 


0 
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14.  The  multiple  time-series  design:  two  similar  groups 

are  measured  repeatedly,  the  treatment  being  administered 
to  only  one  — 


0 

0 


0 

0 


X 


0 

0 


15.  The  recurrent  institutional  cycle  design:  a "patched 

up"  design:  specific  design  features  are  added  to 

control  for  variables  noticed  to  be  threats  to  validity 
once  the  study  is  underway.  An  example  — 


X Ox 


o2  X o3 


16.  Regression-discontinuity  analysis:  an  "award  (scholarship, 

admission  to  university,  etc.)  is  given  to  certain  members 
(chosen  for  high  achievement  for  example)  and  then  only 
that  group  is  measured  to  see  if  the  treatment  (the  award) 
made  a difference: 


Three  tables  are  included  in  the  original  work  to  show  how  each  of  the 
designs  "scores"  on  the  threats  to  both  internal  and  external  validity. 
The  three  are  summarized  in  Table  3.1,  which  has  plus  signs  to  show  the 
strengths  of  the  designs  and  minus  signs  to  indicate  weaknesses  (or 
threats  to  validity) . 

At  this  point,  the  evaluator  has  determined  which  alternative 
research  designs  are  possible  for  his  use.  Now  he  must  decide  how  he 
will  collect  data  from  the  group(s)  to  be  studied.  He  has  two  main 
possibilities:  to  use  data  already  collected  and  available  in  records 

or  to  administer  some  sort  of  measurement  instrument  himself.  The 
former  has  an  advantage  of  being  non-reactive:  that  is,  it  does  not 


selection 
the  award 


those  selected 


of  those  selected 


APPENDIX  C 


CHRONOLOGY  OF  EVENTS 

DATE 

EVENT 

September  26,  1973 

Lester  Rodgers,  Director  of  Office  of 
Regulatory  Standards,  AEC , testifies 
before  Joint  Committe  on  Atomic  Energy, 
U.S.  Congress,  on  the  role  of  regulatory 
standards  in  nuclear  safety. 

February  1,  1974 

Letter  from  Philp  Harter,  Chief, 
Regulatory  Program,  ETIP , to  Lester 
Rodgers  confirming  interest  in  an 
experiment . 

March  21,  1974 

ETIP  completes  a project  plan  for  the 
NRC  experiment. 

April  17,  1974 

Project  plan  is  sent  to  AEC. 

April  25,  1974 

ETIP  submits  further  changes  in  project 
plan  to  AEC. 

April  29,  1974 

AEC  formally  expresses  to  ETIP  interest 
in  a joint  ETIP-AEC  project. 

May  7,  1974 

ETIP  informs  AEC  of  intent  to  proceed 
with  the  project. 

May  16,  1974 

NBS  forwards  project  plan  to  Assistant 
Secretary  for  Science  and  Technology, 
Department  of  Commerce,  for  approval. 

May  23,  1974 

The  Assistant  Secretary  for  Science 
and  Technology,  Department  of  Commerce, 
approves  the  plan. 

June  12,  1974 

ETIP  advances  project  funds  to  AEC. 

August  28,  1974 

Battelle-Northwest  submits  a proposal 
to  the  American  National  Standards  In- 
stitute (ANSI)  for  conducting  the 
NRC  experiment. 

September  10,  1974 

First  meeting  of  N635  committee. 

The  committee  is  organized,  the  content 
of  the  standard  discussed,  and  the  work 
assignments  made. 

DATE 

September  23-24,  1974 
October  15,  1974 

October  29,  1974 

November  1,  1974 

/ 

November  7,  1974 
November  7&8,  1974 

November  26,  1974 
December  18,  1974 

December  31,  1974 
January  9,  1975 
January  15,  1975 

January  24,  1975 
January  25,  1975 


C-2 


EVENTS 

First  meeting  of  N15.28  Committee 

Battelle  begins  work.  Scope  of  standard 
written . 

' 

Harlan  Anderson  submits  a proposal  to 
ANSI  for  conducting  a similar  experiment 
with  another  stnadards  committee. 

Nuclear  Technical  Advisory  Board  (NTAB) 
of  ANSI  meets.  Battelle  presents  descrip- 
tion of  the  experiment.  The  ad-hoc 
NTAB-ETIP  committee  given  responsibility 
to  select  standards  to  developed.  NTAB 
indicates  some  discomfort  with  experimenta 
design . 

NTAB  advises  Battelle  of  three  standards 
to  be  developed. 

Second  meeting  of  N635.  Committee.  The 
approach  to  writing  the  standard  is  dis- 
cussed, assignments  made,  and  a schedule 
developed. 

ANSI  informs  Anderson  of  proposal 
acceptance . 

Battelle  report  #1.  Recruitment  of  N231 
committee  in  progress.  N635  Committee 
already  organized  and  assignments  made. 

Lester  Rodgers,  Director  of  the  Office 
of  Regulatory  Standards,  AEC,  retires. 

ANSI  requests  a representative  from  ETIP 
for  an  ex-officio  position  on  the  ad  hoc 
ANSI-ETIP  committee. 

Executive  order  issued  establishing  the 
Nuclear  Regulatory  Commission  (NRC)  from 
AEC  (effective  January  19) . 

ETIP  reviews  payment  system  due  to  ANSI 
feedback  about  delays. 

TAD  comments  on  experiment  received  in- 
dicating several  weaknesses  of  experiment. 


C-3 

DATE 

EVENT 

January  27-31,  1975 

Third  meeting  of  N635  committe  held  at 
Battelle.  A Draft  1,  Revision  0 is 
produced . 

February  3,  1975 

ETIP  suggests  a representative  for  ad  hoc 
ANSI-ETIP  committee  to  ANSI. 

February  12,  1975 

Second  meeting  of  N15.28  committee. 

Writing  assignments  were  made  and  criteria 
for  the  standard  discussed. 

March  25,  1975 

A project  plan  for  ETIP  Project  #95, 
evaluation  of  the  experiment,  is  completed 
at  ETIP. 

April  7-11,  1975 

Third  meeting  of  N15.28  committee  held 
at  Battelle.  Draft  0,  Revision  3,  is 
produced . 

April  18,  1975 

The  Assistant  Secretary  for  Science  and 
Technology,  Department  of  Commerce, 
approves  the  project  plan  for  ETIP 
Project  #95. 

April  25,  1975 

NRC  is  informed  of  Project  #95  approval 
from  ETIP. 

April  28-Mav  2,  1975 

First  meeting  of  N231  committee  held  at 
Battelle.  The  outline  for  the  draft 
was  revised  and  assignments  made. 

May  6,  1975 

Fourth  meeting  of  N635  committee.  Comments 
on  Draft  1,  Revision  0 reviewed  and 
draft  prepared  for  submission  to  outside 
groups . 

May  13,  1975 

NTAB  meeting  which  was  to  have  discussed 
Project  #95,  but  not  accomplished. 

May  21,  1975 

Fourth  meeting  of  N15.28  committee. 
Comments  on  Draft  0,  Revision  3,  are 
reviewed  and  changes  agreed  to. 

June  16,  1975 

ETIP  funds  for  Project  #95  are  obligated 
to  NRC . 

0 

1 

•U 

DATE 

EVENTS 

Fifth  meeting  of  N15.28  committe.  Draft  0, 
Revision  4 was  discussed.  These  comments 
were  reviewed  until  the  end  of  July. 

July  2,  1975 

ANSI  not  yet  received  the  evaluation  con- 
tract for  Project  #95. 

July  9,  1975 

Delivery  date  for  Project  #47  experiment 
extended  to  December  30,  1975. 

August  1,  1975 

Evaluation  contract  for  Project  #95  still 
in  NRC  contracting  office. 

August  20,  1975 

Chairman  of  ad  hoc  ETIP  committee  changed. 
Ralph  Chalker  resigns  and  Robert  Davidson 
assumes  chairmanship. 

September  1975 

Battelle  submits  final  report  to  ANSI 
for  Project  #47. 

September  1,  1975 

N231  committee  prepares  Draft  1,  Revision  0 
but  does  not  yet  submit  for  outside 
comment . 

December  2,  1975 

NRC  sends  invitation  to  ANSI  for  the  con- 
duct of  Project  #95,  the  evaluation. 

January  23,28,  1976 

ANSI  sends  out  invitation  to  potential 
contractors  for  conduct  of  the  evaluation. 

January  30,  1976 

Three  proposals  submitted  to  ANSI  for 
conduct  of  the  evaluation. 

February  3,  1976 

ETIP  contacts  NRC  to  express  displeasure 
with  the  process  of  contracting  the  eval- 
uation and  need  to  keep  ETIP  informed. 

February  11,  1976 

ANSI  sends  response  to  NRC  invitation 
for  the  Project  #95  evaluation. 

February  27,  1976 

Harlan  Anderson  submits  final  report  on 
C-26  committee  to  ANSI. 

March  26,  1976 

NRC  still  has  not  signed  a contract  with 
ANSI.  New  project  manager  at  ANSI. 

C-5 


DATE  EVENT 


June  14,  1976 

ANSI  sends  second  response  to  NRC  in- 
vitation for  PROJECT  #95  incorporating 
new  components  based  on  ANSI-ETIP  dis- 
cussions . 

July  21,  1976 

NRC  and  ETIP  staff  meet  to  discuss  ex- 
periment and  evaluation.  NRC  express 
desire  that  ETIP  take  control  of  the 
evaluation  effort. 

September  2,  1976 

NRC  formally  requests  that  ETIP  recall 
the  evaluation  fudns  and  manage  the 
effort . 

November  3,  1976 

ETIP  informs  NRC  that  the  funds  obligated 
for  Project  #95  are  to  be  rescinded. 

Novmeber  29,  1976 

Project  #95  deobligated  at  ETIP. 

December  16,  1976 

Project  #47  deobligated  at  ETIP. 
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Experiment  #1:  ANS  2.12  (ANSI  N635)  Guidelines  for  Combining  Natural  Phenomena 

and  Manmade  Hazards  at  Power  Reactor  Sites 


Sponsoring  interest  group:  American  Nuclear  Society  (ANS) 


Variables  tested:  Working  meeting;  draft  of  standard  prepared  and 

distributed  by  executive  secretary;  executive  secretary 
provided;  clerical  and  duplicating  support  provided; 
preliminary  meetings  held;  committee  size  limited  (to  11) 

Sequence  of  events:  — Two  meetings  conducted  before  experiment  started 

— First  meeting  organizational,  assignments 
made  (September,  1974) 

— Second  meeting  used  to  design  overall  writing 
approach;  members  assigned  to  task  groups, 
tentative  schedule  made  (November,  1974) 

— Five-day  working  session  held  in  January,  1975 
— Four  members  not  in  attendance 
— Secretary  compiled  draft  sections  written 
by  individuals  into  working  paper  as 
starting  point 

— Draft  standard  completed  during  course  of  week 
— Agreement  made  to  incorporate  final  day's 
changes  as  the  next  revision 
— Fourth  meeting  held  in  May,  1975  to  review  comments 
on  draft 

— Group  agreed  that  draft  ready  for  review 
outside  the  group 

— Draft  copies  were  sent  for  review  to  committee 
members  and  to  group  of  outside  experts 

Outcomes:  — Standard  cleared  by  appropriate  American  Nuclear 

Society  committee  in  May,  1977 

— Standard  sent  out  for  ballot  by  ANS  members  in  December, 
1977 


— Standard  submitted  to  ANSI  in  October,  1977,  and  sent 
out  for  public  review  and  comment  (concurrent  with 
ANS  membership  review) 

— Writing  group  met  in  February,  1978  to  resolve  com- 
ments; writing  group  response  out  to  ANS  and  ANSI 
by  April,  1978 

— ANS  membership  approved  standard 

— ANSI  approved  as  American  National  Standard  in  July, 
1978 


NRC  received  copy  for  review  and  comment  on  August 
8,  1978 


NRC  sent  comments  back  to  ANSI  committee  on  November  6 
1978. 
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Experiment  #2:  INMM-11  (ANSI  N15.28)  Criteria  and  Standards  for  the 

Certification  of  Nuclear  Materials  Managers 

Sponsoring  interest  group:  Institute  of  Nuclear  Materials  Management  (INMM) 

Variables  tested:  Working  meeting;  outline  of  standard  prepared  and 

distributed;  executive  secretary  provided;  clerical 
and  duplicating  support  provided;  chairman  paid;  travel 
and  living  expenses  paid;  preliminary  meetings  held; 
committee  size  limited  (to  9) 

Sequence  of  events:  — One  meeting  held  before  experiment  started 

— Two-day  meeting  held  September,  1974:  scope 

of  standard  was  drafted;  discussion 

— One-day  meeting  held  in  February,  1975:  discussed 

philisophical  background;  writing  assignments 
made 

— Five-day  working  session  held  in  April,  1975 

— Two  members  not  in  attendance,  and  three 
were  alternates 

— Rudimentary  outline  of  standard  distributed 
to  members  before  meeting  as  starting  point 
— Draft  standard  completed  during  course  of 
week;  some  sections  incomplete 
— Agreement  made  to  incorporate  final  day's 

changes  as  the  third  revision;  chair  agreed  to 
draft  incomplete  sections  after  received 
clean  draft 

— Fourth  meeting  held  in  May,  1975  to  review  comments 
on  draft  (five  attended):  changes  agreed  upon  for 

next  draft 

— Fifth  meeting  held  June,  1975  (seven  attended): 
distributed/discussed , draft  and  agreed  upon 
submission  to  INMM 

— Draft  sent  to  INMM  Executive  Committee  to  review 

— Writing  group  chair  and  other  member  review  Execitive 
Committee  comments,  incorporate  into  revised  draft 
by  end  of  July,  1975 

— Revised  draft  sent  to  writing  group,  designated 
reviewers  and  INMM  Executive  Committee  for  review 

— Draft  completed  by  November,  1975 


Outcomes:  — Standard  submitted  to  ANSI  in  February,  1976 

— Standard  withdrawn  for  revision  in  July,  1976 

Discussion:  When  the  standard  was  sent  out  by  ANSI  for  public  com- 

ment, there  was  so  much  comment  that  the  standard  was 
premature  (because  the  state  of  knowledge  was  insuf- 
ficiently far  advanced)  that  the  standard  was  withdrawn. 


3 


Experiment  #3:  ANS  18.6  (ANSI  N231)  Discharge  of  Thermal  Effluents  Into 

Surface  Waters 


Sponsoring  interest  group:  American  Nuclear  Society  (ANS) 

Variables  tested:  Working  meeting;  draft  of  standard  prepared  and  dis- 

tributed by  chairman;  executive  secretary  provided; 
clerical  and  duplicating  support  provided;  chairman 
paid;  travel  and  living  expenses  paid;  committee  size 
limited  (to  10) 

Sequence  of  events:  — No  meetings  held  before  experiment  started 

— Chairman  drafted  preliminary  draft  and  distributed 
to  members  two  weeks  before  working  meeting;  working 
assignments  made  before  meeting 
— Five-day  working  session  held  in  April,  1975 

— All  members  were  in  attendance,  with  one  an 
alternate 

— First  step  was  major  revision  of  preliminary 
outline 

— Two  draft  revisions  written  and  reviewed 
— Agreement  reached  that  further  work  needed 
before  submission  for  outside  review,  and 
work  assignments  made 

— Assignments  to  members  not  received  as  of  August, 

1975 

Outcomes:  — Draft  never  submitted  for  public  review 

Discussion:  All  environmental  standards  of  ANSI's  Committee  N-19 

(the  ANSI  recipient  committee  for  this  standard)  are 
in  the  process  of  reevaluation,  because  of  problems 
getting  them  through  the  ANSI  public  review  process. 
(Extensive  commentary  on  the  environmental  issues  is 
stopping  approval  of  all  environmentally-related  stan- 
dards.) ANSI  review  of  the  Committee's  standards  has 
thus  been  suspended. 
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